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ABSTRACT 

CNU Online is the computer-managed instructional 
delivery system of Christopher Newport University (CNU) (Newport 
News, Virginia)* Fully functional to deliver online courses since 
fall 1994, CNU Online has undergone a thorough assessment as required 
in its original assessment plan* In the 1994-95 school year, the 
online population did not differ greatly from the university’s 
student population as a whole: it was a diverse group of 
predominantly nont radi t i onal adult learners* The assessment data 
collected were consistent with the view that courses taught online 
are becoming a more established alternative form of instruction at 
CNU* The courses appeared to be as rigorous, or more rigorous, than 
classroom courses, and student satisfaction and performance data, 
where available, were comparable to data from students enrolled in 
classroom courses. Students with prior online experience were more 
likely to choose the online option, and made better progress when 
Lhey did* There was a relatively high withdrawal rate in online 
classes, which improved somewhat in the spring semester. Those who 
persisted in the online experience were usually satisfied, but they 
did report that it was intensive and demanding* The report also 
reveals that a reasonable attempt was made to follow the guidelines 
established for a phased implementation of the online course system* 
Eleven appendixes present additional information about online course 
takers and their achievement* (Contains 4 tables and 5 appendix 
tables*) (SLD) 
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THE 1995 ASSESSMENT REPORT ON GNU ONLINE 



PREFACE TO THE ERIC PRESENTATION 



CNU Online is the computer-managed instructional delivery 
system of Christopher Newport University. Fully functioning to 
deliver wholly online courses since fall, 1994, CNU online has 
just undergone a thorough assessment following the officially- 
approved “Assessment Plan for CNU online.” This following 
document, ”The 1995 Assessment Report on CNU Online” is hereby 
offered for broad dissemination as a helpful contribution to the 
burgeoning field of online instructional delivery. 
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PREFACE 



The 1995 Assessment Report on CNU Online followed the 
organization of the CNU Online Assessment Plan. This plan, 
approved on January 14, 1995, called for a "phased 
implementation," a goal which now has been amply fulfilled. This 
first year has been an intensive learning process, affecting CNU 
Online as a whole; no less, this process has affected assessment 
in particular. Preparation of the current report necessarily 
involved confronting several issues as to how online assessment 
could be most effectively carried out. Indeed, the experience of 
all who have contributed to online assessment will be considered 
in the revision of the assessment plan, scheduled to be completed 
by December 15, 1995. That said, the original plan has served 
reasonably well in bringing about a thorough and provocative 
examination of CNU Online, as a medium of student learning, 
during its first year. 

Organizing this report according to the assessment plan was 
useful in order to keep track of the sheer mass of data. To 
counter some possible fragmentation of perspective, owing to this 
abundance of material, the following abstract will attempt both 
to summarize and integrate the material that follows. What the 
abstract lacks in detailed support is corrected by the report as 
a whole; and what the latter lacks in holistic perspective is 
corrected by the former. Technical report material and tables 
found in the appendices provide even greater support for the 
conclusions of the report. 
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EXECUTIVE SUMMARY 



Highlights 

Extensive assessment data were consistent with the view that 
courses taught online are becoming a more established alternative 
form of instructional delivery at Christopher Newport University. 
Comparative data revealed that: (1) online courses were generally 

as or more rigorous in their expectations than classroom courses; 
(2) students with prior online course experience in the first 
course of a two-course sequence were well-prepared and made 
better progress in the second course than students in the 
corresponding classroom courses; (3) students with prior online 
experiences increasingly chose the online option when given the 
choice; (4) student satisfaction and performance data, where 
available, were comparable to data from students enrolled in 
classroom courses. 

Background- Data 

During 1994-95 the online population was not much different 
from the GNU student population as a whole. It was an extremely 
diverse group of predominantly non-traditional adult learners, 
i.e. beyond traditional college ages of 18-22. There were 
slightly higher ages overall (mean between 29 and 30) and a 
somewhat larger percent (over 90 percent) of classified students. 
On average these students also were somewhat further advanced in 
their progress toward degrees than the student body at large. 

Degree Productivity 

In its first year of operation GNU Online made a modest 
contribution to the institution's degree productivity by helping 
students earn credits required for degrees received. The 
contribution to the BSGA degree, the first targeted degree 
program under GNU Online, was by far the most substantial. These 
conclusions were supported by an examination of May 1995 degree 
recipients and by other evidence regarding increased 
accessibility to courses in the curriculum (see below) . 

Student Retention 



For the first year's evaluation the focus of study was on 
several essential parts of the University-wide issue of student 
retention. One issue was "course retention" or the comparison 
between online and classroom courses in students' persistence, 
after enrollment, through the course to completion. Another was 
"online retention" or the tendency of students to enroll in other 
online courses after having enrolled in one course. 
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I. Course Persistence 

There was a relatively high rate of withdrawal (or low 
persistence) in online classes: 30 and 25 percent in the fall and 
spring semesters, respectively. While the rate improved by 5 
percent from fall to spring, it remained about three times the 
University average. Comparable courses taught in the classroom 
generally had a much lower withdrawal rate as well. 

Several things might explain the high withdrawal rate: 

Students who persisted in online courses were generally 
satisfied and found the experience of comparable value to their 
classroom learning. But they also found the experience to be 
intensive and demanding. The problems of pace and demand were 
exacerbated for students new to CNU Online; they needed to 
familiarize themselves quickly with the system and solve any 
technical problems. Those who have had online experience have an 
advantage and perform better at least initially. 

Other factors affected course persistence. First, students 
who were already enrolled in more than a full load and had 
nothing to lose by dropping one course (due to the billing 
policy) often chose to withdraw from an. online course. Second, 
some withdrawals represented avoidance of a D or F grade. Third, 
some enrollees were relatively uninformed and underprepared for 
online courses and departed as soon as experience showed their 
expectations to be unrealistic. 

II. Online-based Persistence . 

Online students averaged more course enrollments per person 
in the spring semester than in the fall semester. This average 
went from 1.26 to 1.47 enrollments (i.e. courses) per student — a 
statistically significant difference of nearly 20 percent — 
despite the influx of many first time online enrollees. 

Students' greater willingness to devote a substantial portion of 
their load to online courses suggested that students' sources of 
information about online gave them increased confidence. Perhaps 
CNU Online had become more widely known and acknowledged after 
one semester as a viable alternative means of instructional 
delivery. 

The number of fall enrollees returning in the spring was 
substantial relative to the number of fall enrollees 
(approximately 38 percent) . The online enrollment increase did 
not reflect lack of opportunity to enroll in corresponding 
classes offered in the classroom. Such opportunity was available 
in every case. 

Returning students enrolled in more classes in the spring 
than they had in the fall. Their mean enrollments increased from 
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1.36 to 1.60 per student — a statistically significant increase. 

In addition, prior course success was a noteworthy factor. In 
general, returning students had a more prolonged e^osure and a 
more successful experience than non-returning students overall. 

Students who withdrew from an online course, at least in 
fall 1994, were not inclined. to re-enroll in the same course 
online. The other outcomes were about equally divided among 
three other options: dropping or stopping out of the University, 
deferring the -issue of enrolling in the course, or actually 
enrolling in the same or equivalent course for credit in a 
classroom. The low number who re-enrolled in the same online 
course compared to those. who took the standard course suggested 
that students who withdrew tended to have problems with the 
method of course delivery more than with the course per se. 

These findings underscored the importance of helping students to 
be prepared and to know what to expect from an online class. 

Accessibility 

There was evidence that CNU Online increased the typical 
student <s access to parts of the University curriculum. Even 
with the addition of a new residence hall in fall 1994, the 
student population at large remained a commuting population as 
far ae classroom courses was concerned. This important service 
for many students -was easily measurable in the students' savings 
of scarce dollars. Accessibility of University coursework to 
those with mobility problems stemming from a disability was a 
small factor overall. 

Learning Outcomes 

I. General Knowledge and Skills 

CNU Online seeks to develop important general skills 
including (among others) skills related to problem solving, 
synthesis, argumentation, reading, writing and telecommunication. 
Based on students' responses on the CNU Online Instruction 
Evaluation Survey, their perceptions of their achievements in 
these areas were generally positive. In addition, online 
instructors rated student performance and learning in the areas 
related to general skills development substantially higher in 
online than in classroom courses. 

II. Course-specific knowledge and skills 

Students' development of course-related knowledge and skills 
was measured both directly and indirectly. On the Online 
Instruction Evaluation Survey, students responded positively 
(approximate 4 on a 5-point scale); (1) that the course helped to 
develop an appreciation of the historical development of the 
subject matter; (2) that the course helped in learning the 
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vocabulary and concepts of the subject as well as the appropriate 
objectives and values. Similar results were found in an online 
faculty survey. 

There was also some comparative data for four courses (out 
of 12 offered during the semester) . The essential conclusion was 
that the online students' performance was quite comparable to, 
and in some cases exceeded, that of their classroom counterparts. 

III. Course Impacts 

Students' performance in the second course of two-course 
sequences was studied to discover the comparative benefits (if 
measurable) of online course preparation. The data revealed that 
students with online prerequisite experience tended to have an 
advantage in the second online course in a two-course sequence 
over those students who did not have the online prerequisite 
experience. Due to insufficient numbers it could not be 
ascertained whether students with online prerequisite experience 
would perform in a classroom course (the second in a sequence) at 
a level commensurate with students with prerequisite experience 
in the classroom version of the course. 

In lieu of the planned online-to-classroom comparison, 
students' progress within an online two-course sequence was 
compared with other students' progress within the same course 
sequence offered in the classroom. There was an obvious tendency 
for online students to stay the same or improve their grade in 
the second course, in contrast with a trend for more students 
enrolled in the classroom course to earn a lower grade in the 
second course. 

Other student grade data suggested that these results were 
not primaril'^ due to the facilitative effects of being familiar 
with online instruction or with a particular instructor. Rather 
the results were consistent with a view that emphasized the 
educational benefits of close and frequent interaction between 
student and instructor. The online environment provided a 
vehicle in which the instructor regularly and systematically gave 
feedback to develop the critical skills, understandings, problem- 
solving strategies, etc. which are essential to higher levels of 
study . 

Student Satisfaction 



The primary instrument for measuring student satisfaction 
with course delivery was the Online Instruction Evaluation 
Suirvey. This survey was designed to provide measurement which 
was comparable to the standard CNU Instruction Evaluation Survey 
which is used in virtually all classroom courses at CNU. Slight 
revisions to several o" the items were required to make the 
Online Instruction Evaluation Survey appropriate to the online 
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learning environment. The means for both online courses as a 
whole were compared, showing that the lES results were roughly 
the same in most areas. Additional questions were added 
revealing overall student satisfaction concerning issues that 
applied only to online courses. 



Concluding Comments 

Supplementary assessment activities provided further 
evidence that essentially corroborated points made earlier in 
this report. These data were consistent with the view that 
courses taught online are becoming a more established alternative 
form of instructional delivery at Christopher Newport University. 
Students who enrolled in these courses and were unprepared or 
underprepared, or merely curious, found the experience intense 
and difficult. The first part of the course was crucial because, 
due to the intensity of the course, it was easy to fall behind 
and difficult to catch up again. Online courses were generally 
as or more rigorous in their expectations than classroom courses. 
However, the course expectations could differ in important ways 
from classroom courses. They reinforced independent learning, or 
encouraged those who were less independent to become more so. 

They also offered the potential for transportable modular 
presentations such as the Professional Writing Skills Module as 
one challenge to the traditional view of courses as 
compartmentalized, self-contained units. 

Finally, this document served as evidence for assessing the 
adequacy of the current assessment procedures. Modifications and 
adaptations were made during 1994-95; however, this report 
documents that a reasonable attempt was made to follow the 
guidelines of the plan and carry out a "phased implementation" as 
called for. The report also documents problem areas including 
the needs for; 1) more high quality comparative data on course- 
specific student learning; 2) more data on measures of ^eneral 
knowledge and skill development through online instruction. 
Expansion of the scope of assessment studies also would be highly 
desirable but should be weighed in the light of available 
resources. 



THE 1995 ASSESSMENT REPORT ON CNU ONLINE 



Assessment of CNU Online was based on viewing the program as 
integrally related to the CNU institutional context. 

Assessment, so viewed, can be carried out less intrusively and 
with generally greater acceptance and effectiveness. Assessment 
data were also intended to inform and have a positive influence 
on the institution. An extensive study comparing online and 
classroom students may be viewed as part of a larger effort to 
better understand our student clientele. The first step in such 
study is the comparison of demographic characteristics which form 
the context for further discussion of online and classroom 
differences. 



Further, evaluation in this plan takes place within the 
institutional context, deriving information from four distinct 
levels, and in turn potentially informing the institution in a 
number of ways. These levels include (from broadest to 
narrowest) ; > 



Level 1 : 
Level 2 : 
Level 3 : 
Level 4 : 



Online program/ institutional efficiency 
Learning Outcomes 
Teaching effectiveness 

Student satisfaction with course delivery 



Assessment data from several supplementary sources are also 
presented. Finally, the assessment process itself was briefly 
reviewed . 



DEMOGRAPHIC SUMMARY OF ONLINE ENROLLEES 



The study and comparison of online and classroom demographic 
data resulted in the general conclusion that during 1994-95 the 
online population was not much different from the CNU student 
population as a whole. It is an extremely diverse group of 
predominantly adult learners. There is some suggestion of 
slightly higher ages overall and a larger percent of classified 
students. To a slight degree, but only slight, on average these 
students may be further advanced in their progress than the 
student body at large. More advanced students present a good 
match to a number of the higher-level courses offered online. 

Several findings here could not have been anticipated one 
year ago . One of the most important findings was the large 
preponderance of students from the traditional CNU service area — 
a fact which might underscore the need for sophisticated 
marketing efforts at a distance . Another finding was that the 
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number of declared disabilities in this student group was small, 
despite the fact that the original budget initiative had that 
population as one target. 

In the following the specific findings are summarized by 
demographic categories; 

Age; Ages of participants spanned a wide range. The mean 
age was between 29 and 30 years. There was no 
noteworthy change in average age between the fall 1994 
and spring 1995 semester. This average was higher than 
the campus mean but the variation within participant 
ages overshadowed this slight difference. 

Ethnic Status ; 

Approximately 78 percent of the enrollments were White 
and about 17 percent were African-American. A small 
number were Hispanic, Asian or Other. Again, this 
result is quite comparable to the campus-wide breakdown 
of enrollments by ethnic status (e.g., 76 White and 16 
percent African-American.) 

Gender ; 

Males comprised 44 percent of the enrollments and 
females 56 percent. This was comparable to the campus- 
wide division of 39 vs. 61 percent. 

Handicapped and Veteran Statuses ; . 

Less than 1 percent of the enrollments indicated some 
(mostly unspecified) disability on their applications 
for admission. Only one wheel chair-bound enrollee was 
found. Approximately 5 percent indicated veteran 
status. These numbers were typical for GNU students. 

Locations ; 

The majority of students reside in the Virginia 
Peninsula area, i.e. the traditional service area of 
GNU. Approximately 57 percent were from Newport News 
and Hampton in spring 1995. This compared with 47 
percent in fall 1994. A small number were from more 
distant locations — only 2 or less than 1 percent from 
out of state. These results were quite comparable to 
the campus-wide statistics (approximately 52 percent in 
Hampton/Newport News) . 

Status/Level ; 

Students enrolled in GNU Online during 1994-95, were 
overwhelmingly degree-seeking students. Only 8 percent 
of the enrollments were unclassified while 91 percent 
were classified. The small remainder had an audit 
status. The campus-wide statistic was slightly higher 
at 12 percent. In terms of progress towards a degree, 
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approximately 70 percent of all students (including the 
few unclassified) were in the upper classes (junior and 
senior) and approximately 50 percent were seniors. 

Transfer Status . 

The large majority of students (approximately 67 
percent, based on a random sample of 58) had 
transferred credits into CNU upon admission, but this 
result was not very different across the campus. 

Differences by Course : 

For each of the above background characteristics, 
students grouped by course were examined to determine 
whether there were any systematic non-chance 
differences. No such differences were identified. 



LEVEL 1: ONLINE PROGR2VM/INSTITDTIONAL EFFICIENCY 

The assessment under /el 1 attempted to address the 
success in meeting the following five goals: 

1. Degree productivity will be enhanced. 

2. Student retention will be increased. 

3. Accessibility for students with financial and mobility 
problems will be enhanced. 

4. Cost effectiveness and operational efficiency will be 
enhanced . . 

5. Student learning and academic advising will be enhanced as 
shown by analysis of archival information from the digital 
environment. 

Degree Productivity 

In its first year of operation CNU Online made a modest 
contribution to the institution's degree productivity by 
contributing courses required for degrees received. This 
conclusion was supported by an examination of May 1995 degree 
recipients (including those who completed requirements in 
December 1994) . See the table listed as Appendix 1. 

Of the graduating seniors, 37 had been enrolled in wholly 
online courses during 1994-95 i Discounting 5 who were enrolled 
in single courses and withdrew or did not receive a passing 
grade, there were 32 for whom some contribution was made to 
degrees granted. All but three of these completed requirements 
in spring 1995; those three completed their requirements in 
December , 1994. 
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Thus, online courses made a small contribution to several 
degree programs in the program's first year. The contribution to 
the BSGA degree, the first targeted degree program under CNU 
Online, was by far the most substantial. The 14 BSGA graduates 
(all of whom completed degree requirements in spring 1995) used 
some credits from online courses to make progress toward the BSGA 
degree. They represented 39 percent of the 36 graduates who 
received that degree in May 1995. Given that a number of these 
graduates may have completed their requirements prior to spring 
1995, or did not need any of the online courses, this percentage 
is fairly impressive for the first year. Reasonably, it can be 
expected to rise in future semesters. 

Student Retention 



The CNU Online initiative has predicted that online 
instruction would have a long term positive effect on student 
retention at the University. The study of this impact is part of 
the larger study of student retention at the University which is 
ongoing at this writing. For the first year's evaluation, 
therefore, the focus of study was on several essential parts of 
that problem. One issue is "course retention" or the comparison 
between online and classroom courses in student's persistence, 
after enrollment, through the course to completion. Another is 
"online retention" or the tendency of. students to stay with 
online courses after having enrolled in one course. As a 
practical matter, the extensive studies of these issues focused 
on the two previous completed semesters of CNU Online. 

I. Course Persistence 

There was a relatively high rate of withdrawal (or low 
persistence) , in online classes, averaging between 25 and 30 
percent in the fall and spring semesters. A complete 
presentation of the relevant data by course and by semester are 
found in Appendix 2 . The rate improved by approximately 5 
percent from fall to spring, but remained about three times the 
University average. Comparable courses taught in the classroom 
generally had a much lower withdrawal rate as well. 

The phenomenon did not in any obvious way visit particular 
instructors or disciplines more than others; it was a general 
phenomenon of online instruction and was a challenging one to 
interpret. The following is an interpretation that took into 
account a variety of findings. 

(See also Appendix 3.) 

Students who persist in online courses are generally 
satisfied and find the experience of comparable value to their 
learning in the classroom environment. However, they also find 
the experience to be intensive and demanding. They are called 
upon to be in frequent online communication with the instructor 
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and their peers. They find the subject matter with which they 
must deal to be at least as difficult, or more difficult, than 
their counterparts in classrooms. Most students find out quickly 
that in this type of instructional milieu they cannot "get by" if 
they let their wo'k slide. They cannot hope to make up for 
absence from the ' isk by three or four major cram sessions during 
the course. Moreover, a student who falls behind the pace 
quickly becomes obvious to both the instructor and other 
students. The problems of pace and demand also are exacerbated 
for students who are new to CNU Online since they need to become 
familiarized very quickly with the system and solve any technical 
problems. Those who have had online experience have an advantage 
and tend to perform better at least initially. 

The money students have invested in a course is generally an 
important, un-wastable commodity. Since students lose that 
investment by withdrawing, they will try to remain to the end. 
However, there is a category of students who do not stand to lose 
by withdrawing, namely, those who enrolled for 15 or more 
credits. Such students could very easily adopt a strategy which 
says, "Enroll in more courses than you would ordinarily be able 
to handle. If you can handle it, great; if you have to drop a 
course, also great; there is no loss." Such students spend the 
first part of the semester looking over their course schedule to 
determine which, if any, they will have to drop. If they are new 
to CNU On-Line, they will find the experience sufficiently 
different and demanding that the course will become a prime 
candidate for dropping. Their counterparts in the classroom 
courses might adopt the same strategy. However, they will more 
persist in their course since it will not become clear as 
quickly whether they can make it through. 

Students who persisted in and passed an online course have 
had to learn not only course material but also the (for many) 
novel rules of participation. As time proceeds, more students 
will become repeaters and will not have to relearn the rules. 
Also, as CNU On-line becomes better organized to assist new 
students, more students should find a smoother path of adjustment 
to the system. Perhaps some of the modest improvement in course 
persistence from fall to spring was due to such improvements. 
However, because the experience is so intense those who do not 
make quick adjustment, if they do not withdraw, will fall behind 
even faster than they would in the classroom. There at least 
they know the rules. The resulting high failure rate can be a 
problem. 

There may be room ^or other explanations to apply to for 
different students and circumstances. One might be called the 
"migration" hypothesis. When students exercised the option to 
withdraw, a certain numler might have done so to avoid an 
unsatisfactory outcome such as a D, F or the change to an audit 
(AU) in lieu of credit. in effect, the higher online withdrawal 
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rate might be explained as migration of students from one type of 
unsatisfactory category (e.g., D's) to another (W's) . 

• 

However, the data contradicted this hypothesis as a general 
phenomenon. (See Appendix 4.) Students probably did not usually 
exercise the withdrawal option based on their anticipation of a 
worse outcome if they did not. In particular, the consistent 
appearance of more F's in the records of online enrollees is at 
variance with this suggestion. In essence, higher withdrawals 
from online courses accounted for the largest single 
online/classroom difference with respect to less than 
satisfactory grade outcomes. However, in view of the 
consistently smaller number of D's in the online classes, there 
was limited support for the suggestion that some students who 
anticipated receiving less than a C withdrew from the course to 
avoid that outcome. 

II. Online-based Persistence 

This term referred to the tendency for students who had 
taken an online course or courses to persist in taking such 
courses in future terms. There will be at least two 
manifestations of online based persistence: a) the enrollees who 
pass courses in one term will tend to repeat as enrollees in a 
following term, even when the courses they select are offered in 
the classroom as well as online; b) the enrollees who have 
repeated will also tend to take more courses online per student. 

Prior to looking at these data, other findings suggested 
that online-based persistence probably had increased. For 
example, there were indications that students' selections of 
online courses were an expression of their prior satisfaction 
with online courses. Data regarding student enrollments matched 
their self-reported (on the Instruction Evaluation Survey) 
overall satisfaction (see below) with the courses they completed 
and overall preference to take an online course again if given 
the opportunity. 

While enrollments and headcount enrollees both went up 
considerably from fall 1994 to spring 1995 in online classes (264 
and 230 percent respectively) the former outstripped the latter. 
This reflects the fact that students averaged more enrollments 
(i.e. online courses) per person in the later semester. This 
average went from 1.26 to 1.47 enrollments per student — a 
statistically significant difference of nearly 20 percent — 
despite the influx of many first time online enrollees. This 
finding by itself suggested that GNU Online had become more 
widely known and acknowledged after one semester as a viable 
alternative means of instructional delivery. 

However, the finding raised new questions. How many of the 
spring enrollees were returning students who had enrolled in the 
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fall? What was the percent of fall enrollees who returned in the 
spring for one or more online classes? For those returning 
students, were the courses they took offered uniquely online, or 
were they also available at a variety of times in the traditional 
classroom setting? Did the returning students enroll for a 
larger number of online courses than they had in the fall? What 
was the effect of students' relative success as measured by 
grades? 

First, the number of returning students was small relative 
to the total number of students who enrolled in the spring 
(approximately 17 percent) , due to the many newly recruited 
enrollees in the spring. However, the number of returnees was 
substantial relative to the number of fall enrollees 
(approximately 38 percent) . 

Second, the courses offered online in spring 1995 were 
available (by policy) in at least one other section taught in the 
classroom. In some courses the number of classes and the variety 
of times considerably exceeded this minimum. 

Third, returning students did enroll in more classes in the 
spring than they had in the fall. Their mean enrollments 
increased from 1.36 to 1.60- per student — a statistically 
significant increase. 

Fourth, the question of the effect of prior course grades on 
returning enrollments also was addressed. Students who returned 
for the spring, compared to students who enrolled only in the 
fall, tended to have different experiences as shown by their fall 
online grades: 





MULTIPLE COURSES 
TAKEN WITH PASSING 
GRADES IN AT LEAST 
ONE 


NO. WHO RECEIVED AT 
LEAST SOME CREDIT 
FROM THEIR 
ENROLLMENT (S) 


FALL '94 ONLY 


1 OUT OF 64 STUDENTS 
= < 2 PERCENT 


32 OUT OF 64 
STUDENTS =50 
PERCENT 


FALL '94 & SPRING 
'95 


11 OUT OF 41 
STUDENTS = 26.8 
PERCENT 


33 OUT OF 41 
STUDENTS = 80.5 
PERCENT 



The returning students were more likely to have ended the 
term having received some credit for their online enrollments; 
and the returning students were also more likely to have 
attempted multiple courses in the fall and passed at least one. 
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Thus, returning students had a more prolonged exposure and a more 
successful experience than non-returning students overall. 

III. Linking Course Persistence and Online Persistence 

A small study examined the individuals who had withdrawn 
from one or pore CNU Online classes in fall 1994. (See Appendix 
5.) The purpose was to examine whether they followed up in later 
semesters with enrollments in the classes they dropped. Did they 
take the class online or in the classroom, given a choice? This 
study attempted to determine whether the withdrawals were mostly 
related to the course, the method of delivery, or extraneous 
factors. 

The availability of courses, both online and classroom, was 
germane to this study. Since some of the courses were offered 
alternate semesters, it was necessary to expand the scope of the 
inquiry to a third semester (fall 1995) so that a more complete 
cycle of course offerings, both online and classroom, was 
available to the students. 

The following categories were used: 



CATEGORY 


FREQUENCY 


COMMENTS 


I . DROP/ STOP OUT 


9 


A case of 
suspension may be 
included. 


II. ENROLLMENT ONLINE 


2 


One A; the second 
case is ongoing. 


III. ENROLLMENT IN THE 
CLASSROOM 


9 


(Two enrolled in 
languages other 
than Spanish) 


IV. NO RECORD OF COURSE 
ENROLLMENT 


10 


To date the 
requirement is 
unsatisfied 



The data presented above strongly suggested that students 
who withdrew from an online course, at least in fall 1994, were 
not inclined to re-enroll in the same course online. The other 
outcomes were about equally divided among dropping or stopping 
out, deferring the issue of enrolling in the course, or actually 
enrolling in the same or equivalent course for credit in a 
classroom. 

The low number who elected to re-take the online course 
compared to those enrolled in classroom sections suggested that 
students who withdrew tended to have problems with the method of 
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course delivery more than with the course per se. Since the 
course from which the student withdrew always was offered online 
during the two-semester follow-up period, the opportunity was 
available for students. With online courses, work or other 
scheduled activities cannot have been the major issue. 

These findings underscored the importance of helping 
students to be prepared and to know what to expect from an online 
class. There appeared to be an initial period of adjustment to 
online for students like these who had never taken an online 
class before. If they were not properly prepared, they were in 
for a rude awakening. Under those circumstances there was a 
strong likelihood that those students would stay away from online 
classes ever afterwards. 

Another possibility is that the novelty of GNU Online 
interacted with a student's preparedness for online instruction. 
Like any educational innovation, GNU Online has had and will have 
its detractors. Students who had an initial difficulty with an 
online course might have proved quite open to these negative 
judgments. Such evciluations possibly provided a convenient 
"story line" to account for their experiences. If so, one might 
predict that as GNU Online becomes even more established as a 
viable alternative for students, there will be less tendency for 
students to stay away fx .m online courses after one brief 
encounter. In addition to the increasing numbers of multiple 
enrollments by students, a decreasing number of brief, casual 
encounters with online instruction could be a mark of increasing 
program stability in the future. 

Accessibility for students with financial and mobility problems 

The low number of students with disabilities in online 
classes was revealing. It suggested that increasing 
accessibility of University coursework to those with mobility 
problems was a small factor overall. Such students exist in 
small enough numbers within the traditional service area; of 
those, the number who were aware of the program, needed the 
courses or credits, and had access to a computer at home 
apparently was vanishingly small. While the benefit to the few 
was great, increasing their access substantially from close to 
zero, service to the disabled was all but invisible as a general 
feature of the program. The expansion of this service to match 
this part of the original vision will depend upon further 
marketing efforts to reach far beyond the immediate geographical 
area. 



However, there was evidence that GNU Online increased the 
typical student's access to parts of the University curriculum. 
Even with the addition of a new residence hall in fall 1994, the 
student population at large remained a commuting population as 
far as classroom courses was concerned. This was the major 
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breakthrough for many students that is easily measurable in the 
savings of scarce dollars. The locations of students and the 
miles they would need to travel to classroom-taught equivalent 
courses are known data. Making conservative assumptions along 
these lines, such as an average of two round trips per class per 
week, eq[uated to a savings of over 150,000 highway miles in the 
most recent completed semester alone. 

Another dimension of accessibility is time. Scheduling 
conflicts are common for students who work, as most CNU students 
do. (A recent survey of graduating seniors found that 100 
percent were working, 78 percent of those full-time.) Such 
conflicts can add considerable time to a student's academic 
career. Since CNU Online enables students to overcome all such 
conflicts, it is reasonable to anticipate the effects over time 
in the greater efficiency of students' making progress toward the 
degree. Indeed, as demonstrated above after only one year 32 
graduates — 14 of those within a single targeted program — were 
identified for whom CNU Online had contributed to their degree 
attainment. 

Students' responses on three questions on the CNU Online 
Instruction Evaluation Survey confiirmed the above conclusions. 

On a 5-point scale, students responded 4.19, 4.02, and 3.67, 
respectively, to the following questions: 

It is more convenient for me to take this course online 
than in the classroom. 

Online courses are necessary for me to complete an 
undergraduate degree. 

Cost is a factor (scheduling, travel, etc.) in taking 
online courses. 

Cost effectiveness/operational efficiency 

The analysis and report, designed to provide an objective 
test of progress with respect to this goal, will be separately 
prepared and disseminated by June 30, 1996. This analysis will 
take into account the entire fiscal history of CNU Online through 
1994-96. 

Student learning revealed in course archives (message logs) : 

As stated in the CNU Online Assessment Plan: "This area 

represents a serious objective for scholarly research which is 
likely to pay great dividends for practical application. The 
existence of extensive message logs of the CNU Online course 
experience provides a marvelous opportunity. However, this 
interest is long-teirm and will receive lower priority during the 
initial phases of program implementation." 
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An initial pilot study project has begun and probably will 
be presented at ths fall 1995 Virginia Assessment Conference in 
Lynchburg, Virginia, November 8-10, 1995. Professor Roark 
Mulligan of CNU's Department of English has taken the lead on 
this project. 

LEVEL 2: LEARNING OUTCOMES 

The assessment plan called for the generation of direct 
evidence of attainment of student outcomes of two types: first, 
general skills that crossed over many different parts of the 
curriculum, and second, knowledge and skills that are specific to 
the course. Particularly with respect to the latter, comparative 
information (online vs. classroom) was sought wherever possible. 

General Skill Development 

General skills that CNU Online seeks to develop include: 

1. Learning to read, interpret, and critigue written texts 
without oral interpretation. 

2. Learning to write clearly, analytically, and 
persuasively within a conceptual framework. 

3 . Learning to work with peers creating problem-solving 
documents and form cohesive groups. 

4. Learning to use technology. 

5. Learning to be independent learners. 

With regard to these general skills, to date the evidence 
gathered and analyzed are indirect. Students responded to 17 
questions on the CNU Online Instruction Evaluation Survey (lES) 
including several which were unic[uely targeted to measure 
attainment of the above objectives. These questions could not be 
compared with classroom respondents since they were uniquely part 
of the CNU Online lES. (A complete tabular data summary is 
presented in Appendix 6.) Student perceptions of their 
achievements in these areas were generally positive. On a 5- 
point scale, responses on items related to objective number 1 
averaged about 3.79. These responses suggested that students 
believe they have enjoyed benefits with regard to problem solving 
and inferential skills and in reading skills in general. 

Responses related to objective number 2 averaged 4.06, suggesting 
that students believed their writing skills had been developed. 
Regarding objective number 3, students' responses averaged 3.66; 
they perceived growth in the ability to work productively with 
others and resolve controversies, while tolerating others' 
viewpoints. With regard to objective number 4, students clearly 
perceived growth in telecommunication skills and in computer 

17 



O 

ERIC 



20 



software skills (averaging 4.10). No question was directly 
related to obj€\ctive number 5. 

During 'suim er 1995, 12 of 17 online instructors returned a 
faculty questionnaire, part of which was devoted to satisfaction 
with student performance and learning. The survey and summary 
responses for this section are found in Appendix 7 . 

The first eleven questions concentrated upon student 
performance and learning. The mean for these questions was 4.1 
on a scale of 1 (lower than classroom performance) to 5 (higher 
than classroom performance) . Online instructors rated online 
student performance and learning in the areas related to general 
skills development substantially higher than classroom outcomes. 
Means generally were in the four to five range. 

With regard to students becoming independent learners, it is 
noteworthy that faculty uniformly gave the highest rating to 
question number 8: "Overall, do you believe online students 
participate more or less in course discussion than in comparable 
classroom courses?" This finding suggests that online courses 
engender, high levels of involvement and initiative among 
students. This possibility is noteworthy and its implications 
should be explored further. 

The possibilities for analyzing message logs directly to 
measure these areas, particularly development of writing skills, 
are currently being explored. In addition, direct testing of 
critical thinking skills online was addressed and an attempt vas 
made to construct a reasonable instrument for pilot testing 
purposes. However, the effort foundered on the technical issiue 
of creating a scroll-back feature so that students could rev.'.ew 
lengthy passages that filled more than one screen. Furthermore, 
students' patience with extensive testing had already been 
stretched. The technical problem and other issues will be re- 
addressed at a later time. 

Course-specific Knowledge and Skills 

Students' development of course-related knowledge and skills 
was measured both directly and indirectly. On the Online 
Instruction Evaluation Survey, students responded that the course 
helped to develop an appreciation of the historical development 
of the subject matter (3.81); they also responded that the course 
helped in learning the vocabulary and concepts of the subject 
(4.11) as well as the appropriate objectives and values (4.10). 
Similar results were found in the online faculty survey. 

Direct and mostly objective testing was also attempted 
during spring 1995; much of this testing, as directed by the 
assessment plan, was also unit- or module-based. This program 
met with mixed results. The problems encountered were discussed 

18 




2i 



at length with the Provost, the main points of which have been 
appended to this report (Appendix 8) . A primary finding was that 
non-*graded assessment was not being taken seriously by many 
students; therefore an unusual and unacceptable rate of random or 
careless responding was evident in much of the data. This 
finding led to a decision supported by the Provost to move toward 
embedded assessment and to dispense with pretests. (A consultant 
from Behavioral Technologies, Inc., Dr. Edward Smith, had earlier 
recommended that these changes be made in the assessment 
procedures for CNU Online.) 

For these reasons, good quality comparative data, offering 
direct evidence on student learning outcomes specific to courses, 
was in short supply during spring 1995. However, there was some 
comparative data for four courses (out of 12 offered during the 
semester). A summary of these findings is found in Appendix 9. 
The essential conclusion was that the online students' 
performance was quite comparable to, and in some cases excelled, 
that of their classroom counterparts. When students in 
comparable classroom sections were tested using the same or 
similar tests and procedures, the online students performed 
comparably. There was also some evidence that the performance 
was superior when the test format called for writing and a 
critical response. 

Analysis of Prerecmisites or Course Outcomes 

The relative lack of direct test evidence of student 
achievement in specific course learning outcomes led. to adoption 
of an innovative approach toward assessing the effectiveness of 
student learning. This method was proposed in the assessment 
plan as one of the "collateral" methods; however, the results of 
the analysis spawned the realization that this method bears 
directly on the issue of effective student learning of essential 
course-related knowledge and skills. 

Study No . 1 

This study (Appendix 10) examined the question of how 
students performed in spring courses for which the prerequisites 
could be taken in the fall either online or in the classroom. 
There were three courses offered in the fall in both modes which 
were first courses in a two-course sequence where the second 
course was offered in the spring: English 101, English 207G and 
Spanish 101. The corresponding courses offered in the spring 
were English 102, Spanish 102, and English 208G. (The term 
"prerequisite" may be inaccurately applied in some instances; 
e.g., English 207G was not strictly a prerequisite for English 
208G, but it was included here as the first course in a two- 
course sequence.) 
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Part I of this study looked at those who had enrolled in the 
second course online. Given that the second course was taken 
online, how many enrolled in the prerequisite online versus in 
the classroom? Further, how did students perform depending on 
whether they had taken the online or classroom version of the 
prerequisite? 

The data revealed that when sufficient numbers of fall 
online students returned for a second online course in the spring 
to permit a statistical comparison with those whose prerequisites 
were taken in a classroom section, performance in the course 
tends to favor those with the online prerequisite experience. 

However, the interpretation of Part I by itself was 
ambiguous. It was unknown from these results alone how much the 
statistically better perfonnance of the students with the online 
experience could be attributed to experience with the BBS system 
and online pedagogy versus the learning which presumably occurred 
through the online vehicle. Therefore, a Part II was logically 
required because i_ was essential to know how well the students 
with online experience (in the prerequisite) might do in a 
classroom course for which the first course was prerequisite. 
Would these students' performance be comparable with 
traditionally taught students in the same course? 

However, due to the relative newness of GNU Online, the plan 
that was designed to provide an answer to the above question has 
so far failed to provide such an answer . Based on the two 
semesters (94-95) alone, only one person who had taken the 
prerequisite online enrolled for the second course in the 
classroom mode. While this individual clearly was not 
disadvantaged ty the online prerequisite, one person of course 
could not provide enough data to draw a strong conclusion. As 
more terms of online experience accumulate, it will be possible 
to gather sufficient data to test whether students with the 
online experience perform comparably on a second (classroom) 
course for which the online course was prerequisite. 

Nevertheless, while it is not yet possible to test by this 
method how well students have performed in traditional classes 
(as a function of online versus standard classroom 
prerequisites) , another worthwhile observation may be made from 
this study. This observation concerns students' enrollment 
choices. It is noteworthy that of the students with prior online 
experience, if they enrolled at all in the second course in the 
following semester, they tended to choose the online mode again. 
In fact, this pattern, noteworthy as it is, deprived this study 
of the chance to observe online students in a second classroom 
course in sufficient numbers to make an appropriate statistical 
test. 



In sinmaary, students with online prerec[uisite experience 
tended to have an advantage in the second online course in a 
sequence over those students who did not have the online 
experience. Due to insufficient numbers it could not be 
ascertained whether students with online prerequisite experience 
would perform in a classroom course (the second in a sequence) at 
a level commensurate with students with classroom prerequisite 
experience. This lack of numbers# however# was due to an 
apparent tendency for students who enrolled in a second course in 
a sequence to choose the online mode if they had taken and passed 
the first course in that mode. 

Study No. 2: 

Given the lack of numbers necessary to do an online-to- 
classroom comparison of grades (received in the second course as 
a function of the type of prerequisite) , the alternative was to 
compare the progression of classroom-to-classroom students with 
the progression of online-to-online students. Did the latter 
students show progress comparable to the former? 

There were three courses which offered a two-course sequence 
from fall to spring both online and in the traditional setting. 

Of the three only two of them offered the possibility of the 
comparison described above; the third had none of its five 
students from the fall repeating the second course in the spring. 

The first test was related to the transition from Spanish 
101 to 102. For this purpose a particular section of Spanish 102 
which was taught in the classroom in the spring was selected. 

The distribution of grades in this course for this professor was 
fairly typical for this course sequence and thus provided a fair 
basis of comparison. Spring grades were available; 101 grades 
were taken mostly in the previous fall. Ten classroom and nine 
online students grades provided the basis of the test. 

These data suggested a dramatic difference in the 
preparedness of students for the second course as measured in 
terms of grade progression from the first to the second course. 
Consider the results in the following table: 



GRADE PROGRESSION 
TEST FOR SPANISH 
101-102 


GRADE 
EQUAL OR 
INCREASE 


GRADE 

DECREASE 


TOTAL 


CLASSROOM SEQUENCE 


2 


8 


10 


ONLINE SEQUENCE 


7 


2 


9 


TOTAL 


9 


10 
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There was an obvious tendency for online students to stay 
the same or improve their grade in the second course, along with 
an equally obvious opposite trend for the classroom students. 

Due to small numbers a special test called Fisher's Exact Test of 
probability was done. The difference was statistically 
significant at an extremely low probability. 

A second comparison was done for the English 207G to 208G 
sequence. Again one classroom section was selected for 
comparison, and the grades were obtained as before. The 
distribution of grades in this course was fairly typical for this 
course sequence, providing a fair basis of comparison. The 
results of this comparison are shown below: 



GRADE PROGRESSION 
TEST FOR ENGLISH 
207G-208G 


GRADE 
EQUAL OR 
INCREASE 


GRADE 

DECREASE 


TOTAL 


CLASSROOM SEQUENCE 


9 


10 


19 


ONLINE SEQUENCE 


4 


0 


4 


TOTAL 


13 


10 
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These results are more ambiguous. If anything there might 
be the same tendency noted before for online students to continue 
to perform as well or better in the second course; but students 
in the traditional setting are as likely to perform at the same 
or higher level as they are to decrease. Indeed the Fisher's 
Exact Test was inconclusive, showing a probability of .08 — enough 
to suggest there may be something to this comparison but not 
enough to reach the traditional .05 alpha-level of significance. 

A second test using a fair sample of students selected at random, 
who took the course at various times from various professors, 
provided an essentially identical result of .07. 

Overall the results suggested that students' grade progress 
through at least the Spanish course was strong. Grade progress 
through the English course sequence was suggestive but not 
proven. The language course places a strong premium on 
developing the skills and command of syntax and vocabulary. 

These foundations are particularly critical in an introductory 
course such as Spanish 101-102. (Verbal skills are another 
matter that required, and received, additional creativity through 
supplementary methods.) Sequential learning is less evident in 
the literature courses. A professor explained that one course is 
not strictly a prerequisite for the other, and they can be and 
are sometimes taken out of sequence. However, the readings are 
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generally more difficult and the expectations for performance 
higher in 208G. 

Alternative Explanations of Findings 

The potentially important implications of the studies on 
prerequisites suggested the need to address alternative 
explanations as fully as possible with the data available. 

A further question was raised regarding the progression of 
students in sequenced courses such as the three courses studied 
in this phase. Was the progression merely due to familiarity 
with the online system as opposed to the effectiveness of the 
instruction? If there is a general enhancement that has nothing 
to do w*th learning the subject matter, one then would expect to 
find online GPA's increase from one semester to the next among 
those who enrolled in courses in both semester. This possibility 
was tested with 42 individuals who had enrolled in classes in 
both fall 1994 and spring 1995. The various outcomes were as 
follows; 



* (e.g., person went from no grade to grade(s) or had only audits 
in each semester) 

These results could be characterized as a "wash." They 
suggest that a student's grade performance does not automatically 
increase from having had experience with online instruction. 
Therefore the satisfactory grade progression from one semester to 
the next which was noted for the sequenced courses was more 
likely due to the preparation that the students received in the 
earlier semester. 

Other evidence is relevant to the question of whether 
satisfactory progress within online sequenced courses is due to 
extraneous factors besides familiarity with the system. 

Evidence presented above suggested that online experience alone 
could not account for maintenance of grade performance in a 
second course. Another extraneous factor is the ongoing 
relationship with the same instructor. A skeptic, reviewing the 
observed satisfactory grade progress in online courses, might 
suggest that the instructors (who were the same within each 
online course sequence) were hardly disinterested and unbiased; 
having already established a relationship online with their 
continuing students might at some level have predisposed them to 
favor those students. Purely for the sake of argument let us 



ONLINE GPA went up; 
ONLINE GPA went down; 
ONLINE GPA stayed same; 
No test possible* 



13 

14 
10 

5 



TOTAL 



42 



assume this rather cynical view was correct. If so, it would be 
supported by student grades in the second course which were 
noticeably out of line with the students' other general grade 
performance. The suggested empirical check was in fact 
performed. 

Again, the evidence suggested a "wash." In nine cases where 
data were available the second semester online grade was above 
the student's CNU GPA six times and below three times. No 
statistical test was possible on this small group. This test 
merely pointed out that second grades were not remarkably 
discrepant from what one would expect given the students' overall 
grade performance. 

Another alternative suggestion was that it is the type of 
student, not the online preparation, that allows students to 
succeed in online courses. This possibility implies that there 
are differences between online students and classroom students; 
and indeed there are. However, from the earlier section on 
demographic characteristics it appears that the differences are 
not strong or obvious. This fact militates against the 
suggestion. Nonetheless, it might be that some differences such 
as the amount of academic experience and progress towards the 
degree point to a variety of commitment that is crucial interact 
with online preparation to help students with online experience 
to succeed in later courses. 

Discussion of Prerequisites Study 

The results reported above were consistent with a view that 
would emphasize the educational benefits of close and frequent 
interaction between student and instructor. The elements of such 
interaction include the instructor's attention and interest in 
determining what the student understands and does not understand, 
or between what the student can and cannot do. The instructor 
closely monitors and provides continuous feedback to assist in 
developing the critical skills, understandings, problem-solving 
strategies, etc. The online environment provides a vehicle in 
which such interaction can take place regularly and 
systematically. Thus, with online instruction the preparation 
that a first course in a sequence is supposed to provide for a 
more advanced course might have a greater likelihood of 
occurring. 

It is worth emphasizing that the traditional classroom 
course frequently has more ambiguous results in its preparation 
of students for a second or more advanced courses in the 
curriculum, when students' performance is not as closely 
monitored and evaluated, they tend to have a more truncated view 
of the curriculum. They try to survive by completing the next 
assignment, the next test, the next course. When they fall 
behind, they tend to stay behind and fall by the wayside. Poor 
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and hit-or-miss preparation results in lower average grades in 
following courses. (Supporting the argument of this section, the 
researcher found that grades for English 102 tend to be slightly 
but significantly (statistically) lower than for English 101; 
Spanish 102 grades similarly average slightly but significantly 
(statistically) lower than Spanish 101 grades, etc.) Close 
mentorship such as that provided through online instruction can 
reverse this trend toward fragmentation. A key variable is the 
presence of a mentor who assures that students understand and can 
apply what they need to know not only later in the course, but 
throughout the curriculum. 

LEVEL 3: TEACHING EFFECTIVENESS 

The CNU Online Assessment plan called for the peer group for 
each faculty person to review a copy of the message log for the 
courses taught by the candidate. The instructor needed to 
document for the peer group specific examples in the course log 
supporting effectiveness in relation to various criteria relating 
to online teaching effectiveness. For evaluative purposes a 
rating scale was designed to measure instructors' success in 
meeting these criteria. Since the peer review process has not 
yet begun for 1995-96, this portion of the Assessment Plan has 
not been implemented. When implemented, the information 
generated will be useful for internal review and rewarding 
effective online instruction. 

LEVEL 4 : STUDENT SATISFACTION WITH COURSE DELIVERY 

The primary instrument for measuring student satisfaction 
with course delivery was the Online Instruction Evaluation 
Survey. This survey was designed to provide measurement which 
was comparable to the standard CNU Instruction Evaluation Survey 
which is used in virtually all classroom courses at CNU. Slight 
revisions to several of the items were required to make the 
Online Instruction Evaluation Survey appropriate to the online 
learning environment. 

The survey questions and means for both online courses as a 
whole and classroom courses are presented for comparison purposes 
in Appendix 6. The means indicated that the online and classroom 
lES results were roughly the same in most areas. Online results 
were slightly lower, but for an experimental program that demands 
of students both subject matter knowledge and computer skills, 
the means were not surprising and were commensurate with the 
other available data. The means for question number 13, "I found 
this professor to be an effective teacher in this course," were 
nearly identical. 

An additional 17 questions were addressed to online students 
alone. (Some of these have been discussed above under "Learning 
Outcomes.") The mean responses on these items also revealed 
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positive perceptions regarding development of important skills 
through online instruction, such skills included (among others) : 
problem solving, synthesis, argumentation, reading, writing and 
telecommunication. Since these questions were not also asked of 
classroom students a comparison could not be made with online 
sections. The value of these data will increase as they continue 
to be collected in future semesters. 

SUPPLEMENTARY ASSESSMENTS 

Supplementary assessments were of two kinds. First, several 
were carried out which had been proposed in the assessment plan. 
In addition, several others were carried out as circumstances and 
opportunities presented themselves. 

Activities from the Assessment Plan 



Three methods were proposed under the heading of "collateral 
methods" in the assessment plan. The first, the analysis of the 
effectiveness of prerequisites, was reported above. 

A second method involved occasionally inviting external 
consultants to consider assessment-related issues. This was done 
during the past year through inviting Dr. Edward Smith of 
Behavioral Technologies, Inc. Several of his recommendations are 
being followed, including changes in the assessment of student 
learning (referred to above) and the initiation of a pilot study 
of the message logs of one or more online courses. Thus, Dr. 
Smith advised on procedural and methodological issues rather than 
on student outcomes directly. 

Another consultant who has served the CNU assessment program 
in the past. Dr. Robert C. Birney, limited his role to reading 
and commenting on this report. 

A third method involved gathering data from the perspective 
of an individual or individuals who had enrolled in the same 
class, offered both online and traditionally. This activity was 
not carried out as originally conceived due to the lack of staff 
availability for this extremely labor-intensive activity. 

However, as Prof. Ronnie Cohen of the Department of Accounting 
had taken the two-course Spanish setjuence both online (SPAN 101) 
and in the classroom (SPAN 102) , her perspective was both rare 
and potentially helpful. It was rare in that she was the only 
person who could be identified who had enrolled in a fall-to- 
spring online/classroom two-course sequence. 

Professor Cohen's reflections on the comparison between the 
two experiences has been summarized in Appendix 11. She 
identified the comparative benefits of online instruction as 
follows : 
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1. The online experience puts a high premium on students 
becoming independent learners and sustaining 
independent learning. She described this experience as 
"immersion" ... "The kind of performance which is called 
for is much more demanding, and that is good from a 
learning standpoint." 

2. Students find more individualized feedback and 
evaluation in the online course. Feedback is much more 
detailed and individualized than in the face-to-face 
classroom situation where it too often happened that 
some students could avoid participating in more than a 
perfunctory way. 

3. Students benefit from the group work. The benefits 
result from students' role in teaching others, from the 
freer exchange of ideas, from even more individualized 
attention, and from students' greater willingness to 
risk revealing to their peers what they don't know so 
that they can learn. 

4. Another advantage is increasing students' skill and 
comfort level in using computers, including overcoming 
any phobia of computers. 

Turning to actual or potential disadvantages. Professor 
Cohen made the following observations. 

1. It is extremely easy to get behind and not be able to 
catch up. 

2. It is harder to find and exploit opportunities for 
conversation . 



3. Finally, not having personal contact with other 

students is, according to Professor Cohen, in the big 
picture ultimately a disadvantage. Students also need 
to learn to function well in face-to-face situations — 
a goal that a digital learning environment cannot 
substantially help us achieve. 

Dr. Cohen also taught an online course after she had 
completed her role as a student in the Spanish course sequence. 
However, she remained in essential agreement with her earlier 
observations. 

Additional Activities 



The following additional activities also shed some light on 
students and their interactions with CNU Online or tested 
procedures which may provide useful information in the future. 
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While the list appears to be the inevitable catch-all category, 
there were notable themes that emerged, raising new issues and 
suggesting new avenues for further exploration. 

1. Independent Learner Analysis: the Learning Styles 
Inventory 

2 . Inspection of a student petition 

3. Summary report from online telephone counselor, Ms. 

Ruth Kavanaugh 

4. Data relevant to a possible "curiosity effect" 

5. Comparative study of academic "rigor" 

6. Pilot Study of a Professional Writing Skills Module 

The list is by no means exhaustive; however, reporting still 
other activities probably would hot much alter the conclusions as 
regards student outcomes. 

* Independent Learner Analysis 

The following sections are quoted from the discussion 
section of "Learning Style: Attitude and Performance of 

Independent and Dependent Learners," by Doerries, Rieg, Webb and 
Teschner. The paper examined characteristics of dependent and 
independent college students in CNU classroom and online courses 
in spring 1995. 

"These data indicate that more independent learners are 
attracted to the bulletin board based learning environment. 
(12.97% for classroom and 24.80% for online courses) 

It is interesting to note that while a particular type of 
student may be more attracted to one of the learning 
environments, this preference did not manifest itself in terms of 
the grades these students earned. The data are consistent with 
the results from a related study in which it was reported that 
faculty members with independent or dependent learning styles do 
not differentially reward students who are similar or different 
from their own learning style. 

The online environment is expected to attract the more 
independent learner, but furthermore, it is intended to encourage 
and positively reinforce independent learning. The digital 
environment achieves this reinforcement by surrounding the 
student with accessible research material and electronic learning 
tools, rather than by having information presented by the 
lecturer, where the student is in a passive and dependent role. 
Students are also reinforced in the online environment when they 
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engage in independent learning by means of social interaction 
with their peers, since the currency that is valued in peer 
interaction is supplying and interpreting information that is 
useful in solving assigned problems. 

Finally, online environments, even in the relatively simple 
form of bulletin board systems, provide a wide range of functions 
that enable the 'knowledge engineer' who is the teacher, to shape 
the learning behaviors of students.” 

• Inspection of a Student Petition 

A petition related to CNU Online was signed by 460 students 
the spring semester of 1995. To paraphrase the text of the 
petition, it requested the CNU Administration to establish a 
classroom-based class to correspond with every online class. 

The petition asserted its support of that position but did not 
examine or offer debate regarding the current procedures or their 
possible justification. 

The petition was on file in the Office of the Provost. The 
Provost (Dr. Summerville) made a copy available to the Office of 
Assessment from which a comparison was made of the signers at the 
time of the petition (spring 1995) and the rosters of online 
students as of that time (fall 1994 and spring 1995) . 

Two questions motivated the study. First, did signing 
indicate that the signers had direct experience with CNU Online? 
Second, to the degree that signers did have direct experience of 
CNU Online, was that experience negative? Registration data, 
i.e. grades and enrollments, were used exclusively to address 
these questions. 

Of the 460 students who signed the petition, for a small 
number (15 or about 3 percent) it could not be determined whether 
they had taken online courses. Of the remaining 445 students, 
only 19 (or about 4 percent of the remainder) were identified who 
had taken at least one online class. These 19 students had 
enrolled in a total of 32 online classes during the period 
preceding the petition. Thus, the answer to the first question 
was that signing the petition was not indicative of prior direct 
experience with CNU Online. 

Of the 32 online courses taken, the students' grade 
experiences covered a broad range, from A to D (there were no 
F's) and included virtually all other categories such W's and 
I's. However, when the 32 grades were divided between 
satisfactory letter grades (A through C) and all others, there 
were 17 of the former and 15 of the latter. The percentage of 
satisfactory letter grades was far lower than one might expect to 
find from online grades generally. The tentative answer to the 
first question would be that those who had the opportunity to 
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sign and did so tended to have a more negative experience (or 
perhaps briefer or more casual experience) of CNU Online than 
those online students whose names were not found on the petition. 

To a great extent being asked to sign the petition may have 
tapped students' general concern about availability of courses 
and thus enlisted the signers' consent without requiring much 
attention to the matter. However, when the occasional student 
who had some online experience had the opportunity to sign, it 
probably facilitated his or her tendency to sign if the prior 
experience had been less than successful. 

• Summary report from online telephone counselor. Ms. Ruth 

Kavanauah 

Ms. Ruth Kavanaugh is a part time employee for CNU Online. 
Her role has been to contact students who appeared to be having 
difficulties, finding out the source of their difficulties and 
offering helpful suggestions. In late September, 1994 she 
prepared a brief summary report of her findings regarding the 
online students' experiences. 

In brief, Ms. Kavanaugh identified the following main 
problem areas: 

1. Getting started was difficult for many — installing the 
software; learning the system. 

2. Board operations more difficult for Macintosh owners. 

3. Downloading and uploading messages is difficult for 
many. 

4. Students worry about missing deadlines and assignments. 

Ms. Kavanaugh also identified these positives: 

1. Those who had taken previous online courses reported no 
problems using the system. 

2. All reported that. they would take other online courses. 
It is the only way for many to get their degree. 

3. All knew that they could call for help. 

Many changes in services provided have mitigated these 
common difficulties that many students experienced. For example, 
the manual was completely rewritten to be clearer and more user 
friendly. The level of service and support also increased when a 
full-time CNU Online administrator was hired in summer 1995. 
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Data relevant to a possible **curiositv effect ** 



At the outset of the CNU Online program in fall 1994 the 
enrollment patterns were being monitored closely. Since there 
was no experience with online instruction on which to base 
reliable predictions, there was a high degree of uncertainty. 
Contributing to this atmosphere, students who were considering 
online instruction also often had little experience on which to 
base their decisions. Therefore they may have been curious, 
uncertain and tentative. 

A retrospective review of data enrollment patterns from the 
very first days of the program was conducted to test whether the 
data offer any support for this view. Comparing enrollments in 
the eight online classes between September 1, 1995 and after the 
Drop/Add period revealed that the numbers rose from 129 to their 
final official number of 141. This represented a rise of 9.3 
percent. For comparison purposes corresponding classroom 
sections were selected and the numbers compared for the same 
period. The net change was a decline of 2.5 percent. 

These data were surprising in light of the course 
persistence data which revealed just the opposite trend; that 
online students were much more likely to withdraw from the. course 
than their counterparts in the classroom. In retrospect, it now 
appears that much of the early enrollment rise might have been an 
expression of curiosity and exploration since many of the late 
enrollments were ••soft” enrollments that later turned into 
withdrawals. In contrast, the early slight drop in enrollment 
in the corresponding classroom courses did not turn into 
precipitous decline. 

An educational innovation such as CNU Online attracts 
attention and therefore some initial •'soft” enrollments. 

However, enrollments must be coupled with reasonably accurate 
expectations of the time and effort demanded as well as the 
necessity to be an independent learner and overcome any computer- 
related handicaps. Students who were unable to make the 
transition from a more passive mode of learning to CNU Online 
were fairly assured of having an unsatisfying and short-lived 
experience. 

• Comparative study of academic "rigor ” 

A random sample of 50 online students provided a basis for 
testing the comparability of grading online and in comparable 
classroom courses. These 50 students' records provided overall 
GPA's and the online grades. From the latter an online GPA for 
fall 1994 and spring 1995, combined, was calculated. 

The plan of this small study was to compare overall grades 
with online grades, with the student serving as his or her own 
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"control.” That is, comparisons were made between each type of 
grade for each student. The difference (if any) should reflect 
the general level of difficulty for the student of maintaining 
his or her GPA. 

The overall GPA was of course influenced by online grades 
and thus was not a pure measure of grades received by students in 
traditional classrooms. However, since the number of online 
courses taken was generally small relative to the entire student 
record, the overall GPA was used as a convenient and generally 
accurate estimate of students' grades independent of online. (A 
small number who had enrolled only in online classes was excluded 
from this analysis.) 

Out of these 50 students, the online GPA's were higher for 
22 and lower for 28. However, when the students were further 
divided according to those whose online and classroom GPA's were 
quite discrepant (by .50 grade units or more), there were a few 
surprises. There were 30 such individuals out of the 50; of 
these 20 did poorly compared to their GPA derived from classroom 
courses, and 10 did well. There was also a tendency for those 
who did better online to be better prepared academically as shown 
by a higher overall GPA. 

Overall, these results suggested that there was little 
difference between online and classroom courses for these 
students concerning course difficulty as measured by GPA's. 

There is certainly no support for the idea that online courses 
lack rigor, in the strict sense of grading difficulty. 

However, to the extent that students' online GPA's and 
GPA's based on classroom courses were discrepant, there were 
fewer for whom the discrepancy was toward higher grades online. 
Those who flourished online relative to their own academic 
records appeared to be better prepared overall academically. 

This finding also suggested that the online courses possess 
academic rigor. Those who are not well-prepared academically 
tend also not to do well in online courses. Indeed, they tend to 
fall short of their own academic standard of success. 

• Pilot Study of a Professional Writing Skills Module 

During the spring 1995 semester, Dr. Maureen Archer wrote 
and conducted the training and evaluation of the Professional 
Writing skills Module for GNU Online. This module explains and 
gives examples for the components of writing and is designed to 
aid online students in acquiring the essential skills of writing. 
Dr Archer then conducted a pilot study to assess how well the 
Professional Writing Skills Module guidelines were understood and 
how to conduct the evaluations. Dr. Archer worked with several 
professors, gathered pre- and post-tests, trained an evaluator, 
and assessed the results. 
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since only one class participated in the pilot study, and 
the Writing Skills Module guidelines were delayed in being 
forwarded to the students until four weeks into a short 7 . 5 week 
term, the pilot study did not provide an adequate test of the 
effectiveness of this module for the online format. 

Nevertheless, some of the procedures were tested, and on the 
basis of this experience plans were made for a test in the fall 
semester. At this date these plans are on track, professional 
graders have been hired, and there are strong indications that 
the test will be much more satisfactory. 

Concluding Remarks 

The supplementary assessment activities provided further 
evidence that essentially corroborated points made earlier in 
this report. These data are consistent with the view that 
courses taught online are becoming a more established alternative 
form of instructional delivery at Christopher Newport University. 
Students who enrolled in these courses and were unprepared or 
underprepared, or merely curious, found the experience intense 
and difficult. The first part of the course was crucial because, 
due to the intensity of the course, it was easy to fall behind 
and difficult to catch up again. Online courses were generally 
as or more rigorous in their expectations than classroom courses. 
However, the course expectations could differ in important ways 
from classroom courses. They reinforced independent learning, or 
encouraged those who were less independent to become more so. 

They also offered the potential for transportable modular 
presentations such as the Professional Writing Skills Module to 
challenge ways of thinking about courses as compartmentalized, 
self-contained units. 

The potential of online instruction, however, calls for 
renewed efforts to adequately prepare and inform students and 
others. Even with such efforts, "soft” enrollments — i.e., 
enrollments of students who have an inadequate preparation or 
have unrealistic expectations — probably will continue to be a 
factor as they were one year ago. An initial unsatisfactory 
experience can predispose students toward taking a position 
publicly which is (by implication) critical of online education. 
On the other hand, successful experiences predisposed students to 
persevere with online instruction. These findings suggested the 
importance of disseminating accurate and realistic information 
about online instruction. Students need to be better prepared to 
chart their educational careers with the online alternative 
firmly and realistically in mind. 



Finally, this report can be viewed as evidence for assessing 
the adecpaacy of the online assessment process itself. The online 
assessment process was collegial from the outset. The GNU Online 
Assessment Plan was written in a collaborative effort involving 
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the Director of Assessment and Evaluation and Professors Miller, 
Williams, and Teschner. Modifications and adaptations were made 
during 1994-95; however, this report documents that a reasonable 
attempt was made to follow the guidelines of the plan and carry 
out a "phased implementation" as called for by the plan. The 
report also documents where some of the problem areas were and 
suggests the outlines of the next revision, called for by 
December 15, 1995. These areas include at least: 1) more high 
quality comparative data on course-specific student learning; 2) 
more data on measures of general knowledge and skill development 
through online instruction. 

Expansion of the scope of assessment studies also would be 
highly desirable. Issues that need further study include; 1) 
effective teaching strategies by discipline; 2) outcomes uniquely 
or primarily associated with a digital as opposed to a classroom 
environment; 3) comparative general education information; 4) 
other issues as they arise. However, such expansion will depend 
upon the availability of funds and a realistic analysis of work 
load requirements, data access/processing requirements, and staff 
support. Any revision of the CNU Online Assessment Report needs 
to balance what would be desirable against what is currently 
possible or reasonable given resource and staff limitations. 
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APPENDIX 1 



1995 DEGREE RECIPIENTS WHO HAD TAKEN AND PASSED 
AT LEAST ONE GNU ONLINE COURSE DURING 1994-95 



The following table provides the breakdown of 32 May 1995 
degree recipients, who had taken and passed courses online, by 
degree, major and concentration (if applicable) . 



DEGREE 


MAJOR 


CONCENTRATIONS 


NUMBER 


BSGA 


GOVERNMENTAL 


CRIMINAL JUSTICE 


9 




ADMINISTRATION 


LEGAL STUDIES 


3 






PUBLIC MANAGMENT 


2 






TOTAL: 


14 


BSBA 


BUSINESS 


MANAGEMENT 


2 




ADMINISTRATION 


MGMT/REAL EST. 


2 






MGMT INFO SYST 


1 






FINANCE 


1 






TOTAL: 


6 


BA 


HIST(2)/PHIL(1)/ 
SOC WK (1)/ 

POL SCI(l) 


N/A 


5 


BS 


CPSC(3) 

PSYC(2) 

BIOL(l) 


N/A 


6 


BSIS 


INFORMATION 

SCIENCE 


N/A 


1 



ERIC 
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APPENDIX 2 



A REPORT ON GNU ONLINE COURSE PERSISTENCE, 1994-1995 

This study was intended to measure the persistence of 
enrollees in the various online courses and their classroom 
counterparts. A note to report scanners: Your review of this 

report will be aided by reading the bolded portions below. 

Final grade rosters were the primary source of data for the 
analysis. These rosters contain the letter grades (A-F) plus the 
other enrollment statuses and outcomes, including W, P, I, AU, 
and *, brief explanations of which are as follows: 

W = Withdrawal prior to the last announced date allowed for 
withdrawal without grade penalty 

P = Pass, for students exercising a pass-fail option 

I = Incomplete (requiring completion during the next 
semester) 

AU= Audit, or enrollment not for credit 
Methods . 



For purposes of this study, persistence (Ps) is defined as 
the percent of those students who are enrolled in a course (or 
courses) pursuing credit who actually finish the course (s) and 
receive grades (A-P or P) . Audits are not included in the base 
because they are not enrolled for credit. Incompletes also are 
not included because their pursuit of a grade has been deferred 
to the following semester. Non-completion is also termed the 
withdrawal (WD) rate and is simply 100 - Ps. 

From the above definitions, if TOTAL is the number of 
enrollment classification categories for a given course, the 
withdrawal rate, WD, is found by this formula (persistence again 
is 100 minus WD.): 

WD = W/ (TOTAL - AU - I) * 100 

The comparable statistics also can be calculated for 
categories of courses when the number is greater than one, such 
as all classroom courses of a given course number, all courses in 
a given discipline, all courses in a division of the University 
or, finally, all courses at the University. The formula for a 
category of n cases would be: 

WD = SUM (W)/[SUM TOTAL - SUM AU - SUM I] * 100, where sum 
is from 1 to n. This formula in effect averages WD over all 
cases in the category. 

At this writing two semesters (94-95) of the GNU On-line 
experience have been completed, and thus this study focuses on 



the online courses which have been taught during this time. 
Various relevant comparisons will be made below to render these 
results more meaningful . 

Fall. 1994 On-Line Course Persistence 

The following table summarizes the results found thus far by 
course. The table also provides one bottom-line statistic which 
is a composite for all (wholly) online classes taught in fall, 
1994. The reader should be cautioned that a statistic is not 
very meaningful for small enrollments (10 or less) . 



COURSE 


ENROLLMENT 

(CREDIT) 


WD 


Ps 


ENGL 101 - 70 


6 


0 


100 


ENGL 207G- 70 


9 


22 


78 


GOVT 201 - 70 


11 


9 


91 


GOVT 371 - 70 


8 


25 


75 


GOVT 451 - 70 


10 


40 


60 


PHIL 201G- 70 


40 


23 


77 


PHIL 202G- 70 


23 


44 


56 


SPAN 101 - 70 


28 


39 


61 


COMPOSITE 


135 


30 


70 



These . results suggest a rather high rate of withdrawal (or 
low persistence) during fall^ 1994. However, comparisons are 
needed to provide context. 

English ; The numbers enrolled in the online English courses 
may be regarded as too small to be very meaningful. For 
comparison, English 101 (classroom) courses in fall, 1994 
revealed a 5.9 percent WD rate. English 207G (classroom) courses 
showed an 7.3 percent WD rate. The entire English discipline 
curriculum revealed a withdrawal rate of 8.0 percent. 

(Discipline or larger aggregates contain classroom courses, but 
since the percent was small, no attempt was made to remove the 
online contribution to the overall numbers.) 

Government ; The WD rates for the online government courses 
ranged from 9 to 40 percent. Again the comparison courses taught 
in the classroom had generally lower withdrawal rates. Two GOVT 
201 courses showed a combined 8.8 percent WD rate — not different 
from the 9.1 percent for the online courses. However, two GOVT 
371 courses had a 7.4 percent rate as compared to 25.0 percent 
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online. Finally, one classroom-taught GOVT 451 course revealed a 
WD rate of only 5.1 percent compared to 40 percent online. The 
entire Government discipline curriculum had a WD rate of only 8.6 
percent compared to an online aggregate of 24.1 percent. 

Philosophy . The two philosophy courses had WD rates of 22.5 
and 43.5 percent. Five PHIL 201G courses were taught in the 
classroom during fall, 1994, having a combined WD rate of 7.3 
percent9.6, approximately one-third the online rate. Only one 
202G course was taught in the classroom, and its WD rate was only 

2.6 percent. The fall, 1994 Philosophy discipline offerings had 
a WD rate of just 9.3 percent. 

Spanish ; The single online Spanish 101 course, with a WD 
rate of 39.3 percent, was much higher in that statistic than the 
combined number for the 22 classroom Spanish 101 courses which 
were offered in fall, 1994. That combined number was 19.1 
percent, slightly higher than the department's rate of 16. l 
percent . 

For even broader context, the curriculum offerings of the 
Division of Arts and Humanities for fall, 1994 had a WD rate of 

9.7 percent. The corresponding number for the Division of Social 
Science and Professional Studies was just 6.6 percent. (Business 
courses had not been offered online in fall, 1995; thus that 
comparison is not relevant.) Finally, the University wide WD 
rate of 9.8 percent should be cited. 

SUMMARIZING: During the fall of 1994, the first semester in 

which the GNU Online program was in operation, the persistence of 
students who enrolled for credit was substantially LESS than that 
of students who enrolled in comparzdale classroom courses. The WD 
rate for the online courses as a whole was much greater than the 
University-wide trend~in excess of three times that rate. 



Spring, 1995 On-Line Course Persistence 

The following table summarizes the results found for the 
Spring semester by course. The table also provides one bottom- 
line statistic which is a composite for all (wholly) online 
classes taught in fall, 1994. 



COURSE 


ENROLLMENT 

(CREDIT) 


WD 


Ps 


ENGL 102 - 70 


30 


41 


59 


ENGL 208G- 70 


25 


14 


86 


GOVT 201 - 70 


23 


45 


55 



5 




43 



GOVT 202 “ 72 


35 


20 


80 


GOVT 331 - 72 


36 


19 


81 


GOVT 355 - 70 


39 


33 


67 


GOVT 358 - 71 


27 


20 


80 


PHIL 201G- 71 


46 


24 


76 


PHIL 202G- 72 


62 


17 


83 


PHIL 395P- 70 


7 


25 


75 


SOCL 201G- 70 


16 


25 


75 


SPAN 102 - 70 


26 


23 


77 


COMPOSITE 


372 


25 


75 



These results again show a high rate of withdrawal (or low 
persistence ) g but these numbers are improved by approximately 5 
percent, one administrator looked for improvement on the basis 
of better organization in the spring; and indeed, there was a 
small improvement. Again, comparisons by course and discipline 
are useful. 

English ; ENGL102-70 had a WD rate of 40.7 percent as 
compared to 8.6 percent for the classroom version of ENGL102. 

The WD rate of ENGL208G-70 was 13.5 percent; in comparison, the 
WD rate for the other two ENGL208G courses was 7.8 percent. The 
entire English discipline curriculum had a withdrawal rate of 
only 8.7 percent. 

Government ; There were five GOVT courses offered wholly 
online in spring, 1995. For every one of the five comparable 
classroom courses, the class offered under that designation had a 
lower WD rate than the online version of the course. The 
composite comparison for GOVT was: a WD rate of 24.1 percent for 
the online courses compared with only 7.7 percent for the 
comparable classroom courses. The GOVT discipline as a whole had 
a WD rate of just 10.1 percent. 

Philosophy : Enrollment in PHIL395 was extremely small and 

will not be separately considered. PHIL 201G-71 and 202G-72 both 
had higher WD rates than the comparable classes under each 
designation taught in the classroom. The composite online rate 
of withdrawals was 20.4 percent as compared with the combined 
rate for the equivalent classroom courses of only 5.1 percent. 

The PHIL discipline WD rate was- only 8.7 percent. 

Sociology : SOCL201G-70 had a WD rate of 25.0 percent 

compared with only 4.1 percent for the two other SOCL201G 



6 



O 



44 



classes. The SOCL discipline WD rate was just 7.6 percent. 

Spanish : SPAN102-70 had a WD rate of 23.1 percent compared 

with 12.6 percent for classroom-based SPAN102 courses. The WD 
rate for Modern Languages was 9.9 percent. 



For broader context, the curriculum offerings of the 
Division of Arts and Humanities for spring, 1995 had a WD rate of 
8.5 percent. The corresponding number for the Division of Social 
Science and Professional Studies was just 6.6 percent. (Still no 
Business courses been offered online in spring, 1995.) Finally, 
the University wide WD rate of 8.4 percent should be cited. 

SUMMARIZING: During the spring of 1995, the second semester 

of the CMU Online program, the persistence of students who 
enrolled for credit was still substantially lower than that of 
students who enrolled in comparable classroom-based courses. The 
WD rate for the online courses as a whole was much greater than 
the University-wide trend— still slightly more than three times 
that rate. However, there was a slight improvement in the rate 
from fall to spring — from 30 to 25 percent. 
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APPENDIX 3: MEMORANDUM ON COURSE PERSISTENCE 



TO: 



RE: 

DATE 



FROM 



Provost, Executive Assistant to the President, BACUP 
Self-Study Steering Committee, Student Assessment 
Committee, Academic Deans, CNU On-Line staff and 
teachers 

Dennis R. Ridley 

Director of Assessment and Evaluation 
Course Persistence, Online vs. Offline 
August 4, 1995 



I have been encouraged to make the findings on this issue 
widely available. However, rather than sending out another 
report like the last one (a revised version of which has been 
completed and is available), I will summarize the results along 
with some related points. Finally, I will offer an 
interpretation of what is going on to produce these results. 

I trust that most will find this updated report stimulating, 
but please remember that it is only preliminary. The reader is 
also encouraged to remember that the measure of persistence is 
only one among a number of measures that make up assessment of 
online instruction. 

Summary of findings related to persistence 

1. There was a relatively a high rate of withdrawal (or 
low persistence) , in online classes, averaging between 25 and 30 
percent in the fall and spring semesters. The rate improved by 
approximately 5 percent from fall to spring, but remained about 
three times the University average. Comparable courses taught in 
the classroom generally had a much lower withdrawal rate as well. 

2 . The phenomenon does not in any obvious way visit 
particular instructors or disciplines more than others; it is a 
general phenomenon of online instruction. 

3. Persisters in online courses (those who remain and 
receive a grade) compared to persisters in other courses were not 
generally the better or more successful students overall. 

However, persisters in online courses (excluding failures) 
averaged consistently higher gradewise than their counterparts 
(classroom course persisters sans failures) in the comparable 
courses. This finding was fairly consistent across courses. 

4. Among persisters, the rate of failures was also 
consistently higher in the online courses than in the counterpart 
classroom courses. 

5. Course success (defined as persisting and passing) was 
examined in two spring online courses where some took the 



prerequisite course online and others offline. This comparison 
showed significantly (statistically) better success for those who 
had the online experience for the prerequisite or first course in 
the sequence. 

6. Withdrawals from online courses were more likely to 
occur when the student already has a "full plate", i.e., 15 or 
more credits. (In that situation a withdrawal from a 3 credit 
course did . not involve a monetary loss since more than 12 credits 
can be taken for the same tuition.) However, what is suspected 
(but untested as yet) is that overload is not as strong as a 
deteirminer of withdrawals from the classroom courses. 

7. Instructor Evaluation Survey results comparing online 
and classroom courses are available and are generally comparable 
(that is a subject for a different memorandum) . One particular 
question which online students tend to agree with more highly 
than their classroom counterparts is, "The subject matter of this 
course is difficult." 

Interpretation of online persistence 

Students who persist in online courses are generally 
satisfied and find the experience of comparable value to their 
learning in the classroom environment. However, they also find 
the experience to be intensive and demanding. They are called 
upon to be in frequent online communication with the instructor 
and their peers. They find the subject matter with which they 
must deal to be at least as difficult, or more difficult, than 
their counterparts in classrooms. Most students find out quickly 
that in this type of instructional milieu they cannot get by if 
they let their work slide. They cannot hope to make up for 
absence from the task by three or four major cram sessions during 
the course. Moreover, a student who falls behind the pace 
quickly becomes obvious to both the instructor and other 
students. The problems of pace and demand also are exacerbated 
for students who are new to CNU On-Line since they need to become 
familiarized very quickly with the system and solve any technical 
problems. Those who have had online experience have an 
advantage and tend to perform better. 

The money students have invested in a course is generally an 
important, un-wastable commodity. Since most students stand to 
lose that investment by withdrawing, they will try to remain to 
the end. However, there is a category of students who do not 
stand to lose by withdrawing, namely, those who enrolled for say, 
15-18 credits. Such students could very easily adopt a strategy 
which says, "Enroll in more courses than you would ordinarily be 
able to handle. If you can handle it, great; if you have to drop 
a course, also great; there is no loss." Such students spend the 
first part of the semester looking over their course schedule to 
determine which, if any, they will have to drop. If they are new 
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to CNU On-Line, they will find the experience sufficiently 
different and demanding that the course will become a prime 
candidate for dropping. Their offline counterparts might have 
the same strategy. However, they will more likely persist in 
their course since it will not become clear as quickly whether 
they can make it through. 

Students who persisted in and passed an online course have 
had to learn not only course material but also the (for many) 
novel ruiec of participation. As time proceeds, more students 
will become repeaters and will not have to relearn the rules. 
Also, as CNU On-Line becomes better organized to assist new 
students, more students should find a smoother path of adjustment 
to the system. Perhaps some of the modest improvement in course 
persistence from fall to spring was due to such improvements. 
However, because the experience is so intense those who do not 
make quick adjustment, if they do not withdraw, will fall behind 
even faster than they would in the classroom. There at least 
they know the rules. The resulting high failure rate can be a 
problem. 
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APPENDIX 4 : ONLINE COURSE PERSISTENCE POLLOW-UP STUDY: 

THE "MIGRATION" HYPOTHESIS 

The higher withdrawal rate among online than traditional 
classroom students suggested another possible explanation. When 
students exercised the option to withdraw, a certain number might 
have done so to avoid an unsatisfactory outcome such as a D, F or 
the change to an audit (AU) in lieu of credit. In effect, the 
higher online withdrawal rate might be explained as migration of 
students from one type of unsatisfactory category (i.e., D's) to 
another (W's). This possibility would suggest that when these 
other grade categories are added to the W category, the resulting 
percent of all enrollments would be much closer to equal for 
online and classroom students. Thus, overall "unsatisfactory 
quotients" were found by dividing the course enrollments by the 
various low- or non-grade categories for the course combined. 

["pii (for Passing) was excluded because it is a satisfactory 
outcome; "I" (Incomplete) was also excluded because it is 
deferral of a grade effort until a later time.] The quotient was 
thus the percent of total enrollments in a class, or group of 
classes, in which the outcome was less satisfactory than an A 
through C grade. 

For the fall '94 and spring '95 semesters these quotients 
were found for each online class and for the same classroom 
section or sections (combined) . The differences (in percent of 
"unsatisfactory" grades) between the online courses and their 
classroom equivalents were also found. In order to arrive at a 
composite number for an entire semester, weighted averages were 
found. The purpose of this procedure was assure that the 
contribution of a deviation to the summary statistic for the 
semester was in direct proportion to the size of the online 
class. Thus a Philosophy topics class, for example, would have a 
small weight and the heavily enrolled Philosophy 201G (a liberal 
studies distribution course) would have a much larger weight 
applied to its deviation score. The result for the fall was a 
deviation of approximately 17 percentage points. Thus, on 
average the online classes were considerably higher than their 
classroom comparisons in the "unsatisfactory" measure. The 
corresponding result for the spring was about 9 percent. Thus, 
the results. for the two semesters were not close together on this 
composite statistic. (However, the manner in which they differed 
is revealing as will be discussed below.) 

For this result to be more meaningful, it had to be compared 
with the overall withdrawal rate deviation, calculated in the 
parallel way. The withdrawal rates for each course (classroom or 
online) or group of classroom courses were found practically as 
they had been before. One inconsequential difference was that 
audits were added to the denominator so that the withdrawal and 
"unsatisfactory" measures would be exactly comparable. The 
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weighted average withdrawal deviations were found for both fall 
'94 and spring '95. For the fall that average was approximately 
12 percentage points. This reveals that online students tended 
to withdraw more frequently by this measure of percentage 
difference. Similarly, the parallel finding for spring was 
approximately 11 percentage points. These two results are 
consistent. 

Comparison between the mean withdrawal and ’’unsatisfactory" 
deviations (between online and classroom) was suggestive. It 
suggested that the elevation of one ’’unsatisfactory’’ outcome 
(withdrawal) for online courses was not generally compensated for 
by the elevation of other indicators for the equivalent courses 
offered in the classroom. While the results were fairly 
dissimilar between the two semesters. However, both results were 
difficult to reconcile with a hypothesis that withdrawal was 
largely due to migration toward withdrawals from anticipated less 
satisfactory outcomes. 

Some further insight into, this result was provided by 
identifying the contribution of each type of outcome (D, F, AU 
and W) to the results. These contributions are best thought of 
as components which add up to the totals given above. Again, the 
totals are summary statistics representing the overall difference 
between online and comparable classroom courses. These results 
are summarized in the following table: 
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GRADE OUTCOMES 


FALL '94 

DIFFERENCE 

VALUE 


SPRING '95 
DIFFERENCE 
VALUE 


COMMENTS 


D-RATE COMPONENT 


.06 


-4.35 


D's were fewer 
online in most 
classes. The fall 
value is positive 
only because of a 
huge contribution 
of one course. 


F-RATE COMPONENT 


4.02 


1.92 


F's were higher 
online in most 
classes both terms; 
less so in the 
spring 


AU-RATE COMPONENT 


1.05 


0.42 


AU's tended to be 
higher online but 
made only a small 
difference 


W-RATE COMPONENT 


12.22 


10.89 


The difference 
(online vs. 
classroom) was 
consistent 


"UNSATISFACTORY " 
INDEX 

(ONLINE/ CLASSROOM 
DIFFERENCE) 

( D+F+AU+W) /TOTAL 


17.35 


8.88 


Note that the 
components above 
add up to these 
summary statistics. 



Discussion and Conclusions 



The table above contradicts the proposed hypothesis as a 
general phenomenon. Students did not generally exercise 
the withdrawal option based on their anticipation of a worse 
outcome if they did not. In particular, the consistent 
appearance of more F's in the records of online enrollees is at 
variance with this suggestion. In essence, higher withdrawals 
from, online courses account for the largest single part of the 
const^cted "unsatisfactory" difference index. Indeed, their 
contribution is larger than the overall "unsatisfactory" 
difference rate in the spring '95 semester. In view of the 
consistently smaller number of D's in the online classes, there 
is limited support for the suggestion that some students who 
anticipated receiving less than a C withdrew from the course. 



APPENDIX 5 



LINKING COURSE PERSISTENCE AND ONLINE PERSISTENCE 

This small study examined the individuals who had withdrawn 
from one or more cnu online classes in fall '94. The purpose was 
to examine whether they followed up in later semesters with 
enrollments in the classes they dropped. Did they take the class 
online, or in the classroom, given the choice? This study 
attempted to determine whether the withdrawals were mostly 
related to the course, the method of delivery, or extraneous 
factors. 

The availability of courses, both online and classroom, was 
germane to this study. Since some of the courses were offered 
alternate semesters, it was necessary to expand the scope of the 
inquiry to a third semester (fall '95) so that a more complete 
cycle of course offerings, both online and classroom, was 
available to the students. 

For the purposes of this study, student records were 
examined for a total of 34 students who had withdrawn from at 
least one course. Thirty of the 34 had withdrawn from 1 course 
and 4 had withdrawn from 2 courses. In the interests of clarity, 
the 4 with two withdrawals will be discussed in a separate small 
section. This will permit presentation for the majority (i.e., 
30) to follow and classify the outcomes of a single course per 
student . 

After examining the records of the 30 students, the 
following categories were devised to cover (with minor 
discrepancies) all of the cases: 

I. Dropping/stopping out: Since withdrawing from the online 

course (and perhaps other courses), the students' records 
revealed no further enrollments at the University. None had 
graduated . 

II. Follow-up enrollment in an equivalent online class: The 
student enrolled in an online class in one of the two 
following semesters. 

III. Follow-up enrollment in an equivalent classroom course: The 
student enrolled in an classroom section in one of the two 
following semesters. 

IV. No record of any follow-up enrol Imenrs: Although the 

student ostensibly remained enrolled at the University, 
there is no evidence that he or she attempted to take the 
same course again, either online or in the classroom. 
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CATEGORY 


FREQUENCY 


COMMENTS 


I . DROP/STOP OUT 


9 


A case or two of 
suspension may be 
included. 


II. ENROLLMENT ONLINE 


2 


One A; the second 
is ongoing. 


III. ENROLLMENT IN THE 
CLASSROOM 


9 


(2 enrolled in 
languages other 
than Spanish) 


IV. NO COURSE ENROLUIENT 


10 


Enrolled in other 
classes. 



Regarding the four cases of withdrawals from two online 
courses, the same categories could be applied. Two could be 
classified as dropping or stopping out; one had no record of 
enrollment, and the fourth enrolled in both courses online again 
during the following semester (but failed both) . 

Discussion and Conclusion 

The data presented above strongly suggested that students 
who withdrew from an online course, at least in fall * 94 , were 
not inclined to re-enroll in the same course online. The other 
outcomes were about equally divided among dropping or stopping 
out, deferring the issue of enrolling in the course, or actually 
enrolling in the same or equivalent course for credit in a 
classroom. 

The low number who elected to re-take the course compared to 
those who took the course in the classroom suggested that 
students who withdrew tended to have problems with the method of 
course delivery more than with the course per se. Since the 
course from which the student withdrew always was offered online 
during the two-semester follow-up period, the opportunity was 
generally available for students. With online courses, work or 
other schedules cannot have been the issue. 

These findings underscore the importance of helping students 
to be prepared and to know what to expect from an online class. 
There appears to be an initial period of adjustment to online for 
students like these who had lUever taken an online class before. 

If they were not properly prepared, they were in for a rude 
awakening. Under those circumstances there is a strong 
likelihood that those students will stay away from online classes 
ever afterwards. 

Another possibility is that the novelty of CNU Online 
interacted with a student ^s preparedness for online instruction. 
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Like any educational innovation, CNU Online has had and will have 
its strong detractors. Students who had an initial difficulty 
with an online course might have proved quite open to these 
negative judgments. Such evaluations possibly provided a 
convenient "story line" to account for their experiences, if so, 
one might predict that as CNU Online becomes even more 
established as a viable alternative for students, there will be 
less tendency for students to stay away from online courses after 
one brief encounter. In addition to the increasing numbers of 
multiple enrollments by students, a decreasing number of brief, 
casual encounters with online instruction could be a mark of 
increasing program stability in the future. 

Follow-up interviews with as many of these students as 
possible would go a long way towards testing the conclusions of 
this section. 
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APPENDIX 6 



GNU INSTRUCTION EVALUATION SURVEY MEAN RESPONSES 
ONLINE AND CLASSROOM STUDENTS, 

FALL 1994, SPRING 1995, AND SUMMER 1995 



Item 
= 3; 


response scale: "Strongly Agree” = 5; 
"Disagree" = 2 ; "Strongly Disagree" = 


"Agree" = 

1. 


4; "Neutral" 




QUESTION 


CLASSROOM 

Spring 

1995 


ONLINE 

Spring 

1995 


ONLINE 
Fall 1994/ 
Spring '95 
Summer '95 


1. 


The course the instructor 
presented was consistent 
with the course 
description in the 
Catalogue. 


4.59 


4.25 


4.31 


2. 


The instructor came to 
class well -prepared. 

(The instructor's messages 
showed evidence of careful 
thought and preparation.) 


4.65 


4.32 


4.45 


3. 


The instructor's grading 4.52 

policies were clearly 
explained early in the term. 


4.22 


4.24 


4. 


The instructor's grading 
procedures were fair. 


4.41 


4 . 17 


4.26 


5. 


Graded assignments and 
tests were returned in a 
timely fashion. 


4.53 


4.22 


4.25 


6. 


Tests covered knowledge, 
application, or reasoning 
that could be expected on 
the basis of course 
content . 


4.35 


4.21 


4.23 


7. 


The instructor maintained 


4.33 


4.35 


4.40 



regular office hours and 
encouraged students to seek 
help when needed. (The 
instructor encouraged 
students to seek help when 
needed and was readily 
accessible when he or she did so.) 
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8. 


The instructor demonstrated 
command of the subject 
matter of the course. 


4.63 


4.55 


4.61 


9. 


The instructor's presenta- 
tion of the course material 
was clear and understandable. 
(The instructor's messages to 
me and to the class made the 
course material clear and 
understandable.) 


4.25 


4.10 


4.04 


10. 


The instructor demonstrated 
interest and enthusiasm for 
the sxibject matter. 


4.61 


4.40 


4.52 


11. 


The instructor was 
intellectually motivating 
and stimulated learning. 


4.23 


4.10 


4.25 


12. 


The instructor met classes 
on time and for adequate 
duration. (The instructor 
consistently devoted the 
time necessary to make this 
course a valuable learning 
experience for me . ) 


4.72 


4.14 


4.19 


13. 


I found this professor to 
be an effective teacher 
in this course. 


4.34 


4.26 


4.33 


14. 


The subject matter of this 
course is difficult. 


3.49 


3.80 


3.84 


15. 


The subject matter of this 
course is interesting. 


3.97 


3.86 


3.94 


16. 


The subject matter of this 
course is a valuable part 
of my education. 


4.01 


3.72 


3.64 



AODITIOKAL QUESTIONS UNIQUELY FOR ONLINE STUDENTS, Fall 1994, 
Spring 1995, and svunmer 1995 (Questions 17-33) (Item response 
scale same as above.) 

17. The course developed my problem 3.44 

solving skills. 

18. The course enabled me to draw 3.91 

reasonable inferences from 
observations. 
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19. The course developed my ability to inte- 3.97 
grate and synthesize information. 



20. The course developed my ability 3.93 

to use facts to support opinion. 

21. The course developed my ability 3.81 

to appreciate the historical 
development of the subject 
matter addressed in the course. 

22. The course developed my tolerance 3.85 

for other viewpoints. 

23. The course developed my ability 3.64 

to work productively with others. 

24. The course developed my ability to 3.50 

resolve controversies. 

25. The course helped me learn the 4.11 

vocabulary and concepts of the 
sub j ect . 

26. The course helped me learn the 4.10 

objectives and values of the 
sub j ect . 

27. The course developed my reading 3.68 

skills. 

28. The course developed my writing 4.06 

skills. 

29. The course developed my 4.17 

telecommunication skills. 

30. The course developed my computer 4.03 

software skills. 

31. It is more convenient for me to 4.19 

take this course online than in 
the classroom. 

32. Online courses are necessary for 4.02 

me to complete an undergraduate 
degree . 

33. Cost is a factor (scheduling, 3.67 

travel, etc.) in taking online 
courses . 
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APPENDIX 7 



ONLINE FACULTY QUESTIONNAIRE MEANS AND ANALYSIS-SUMMER 1995 

Online faculty satisfaction levels with student performance and Mean 
learning (5-point scale used where Higher > 5 and Lower > 1) Response 

1. How do you rate the performance of cooperative learning groups 

online with classroom group work? 4.0 

2. How do you rate the development of student writing skills online 

in comparison to comparable classroom courses? 4.3 

3. How do you rate the development of student independent reading 

comprehension skills online in comparison to comparable classroom 
courses? 4.1 

4. How do you rate student learning levels of basic course 

vocabulary and concepts online in comparison to comparable 
classroom courses? 3.8 

5. How do you rate the development of student problem solving 

skills online in comparison to comparable classroom courses? 4.4 

6. How do you rate the development of student integration and 

synthesis of information skills online in comparison to comparable 
classroom courses? 4.2 

7. How do you rate the level of student tolerance for other 
viewpoints online in comparison to comparable classroom courses? 4.0 

8. Overall, do you believe online students participate more or 

less in course discussion than in comparable classroom courses? 5.0 

9. Overall, do you believe students learn more or less course 

subject matter online than in comparable classroom courses? 4.3 

10. How do you rate the quality of student-to-f acul ty interaction 

online in comparison to comparable classroom courses? 3.7 

11. How do you rate your satisfaction with the online medium? 3.6 

Assessment plan implementation 

The measures for determining level of implementation of the 
assessment plan are; pre-test and post-test as measures of learning 
outcomes, incorporation of critical thinking skills, use of cooperative 
learning among peers, techniques of data retrieval and analysis, and 
participatory interaction. 

12. As a designer of an online course, I found it useful to use 
pre-test and post-test modules or units in order to measure 



learning outcomes of students. 2.0 

13. As a designer of an online course, I created a course that 

taught students skills of comprehension, analysis, synthesis, 
application, and evaluation of the subject matter. 4.3 

14. As a designer of an online course, I used student work groups 

to create learning environments where students learned to 
negotiate differences among themselves while creating group work 
products. 4.3 

15. As a designer of an online course, I instructed students how 

to retrieve, analyze, and interpret data using computer-managed 
resources. 3.5 



16. As a designer of an online course, I managed the development 
of a message log in which student messages were characterized by 
contextual coherence, thematic development, convergent and 
divergent thinking, conceptual analysis, and civility and 
tolerance. 4.2 

Faculty Training 

The following are measures of the results of faculty training for 
online teaching: participant interactivity, informal /formal message 

distinction, formal /informal message quality, uploading/downloading 
techniques, offline mail reader procedures, and techniques for work 
group procedures. 



17. As an online faculty member, I was made aware of how to 

organize subject matter material to enhance interactivity between 
participants. 3.9 

18. As an online faculty member, I was introduced to the 

distinction between informal and formal messages. 4.1 

19. As an online faculty member. I was made aware of criteria for 

measuring the quality of informal and formal messages. 3.9 

20. As an online faculty member, I was made aware of how to 
upload and download messages from a word processor to CNU 

ONLINE. 3.8 

21. As an online faculty member, I was introduced to how to use a 

mail reader to download many messages at one time; answer the 
messages offline; and then upload them in one operation. 3.4 



22. As an online faculty member, I was introduced to how to 
organize students into groups for purposes of constructing group 
work products online. 
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Online Faculty Questionnaire Analysis (by Professors Williams and Miller) 



Out of 17 online instructors during summer 1995, 17 returned the 
questionnaire. Two questionnaires were not fully completed. 

The first eleven questions concentrated upon online faculty satisfaction 
levels with student performance and learning. The mean for these questions is 
4.1. on a scale of 1 (lower than classroom performance) to 5 (higher than 
classroom performance). Online instructors rate online student performance 
and learning substantially higher than classroom outcomes. Every mean was in 
the four to five range with the exception of student learning levels of basic 
course vocabulary and concepts. However, even the mean for this question 
(3.8) was above that of the classroom. 

The second part of the questionnaire (questions 12-16) focused on the 
implementation of the online assessment plan. The overall mean for these five 
questions is 3.7. The only question to fall into the two range (2.0) was, "As 
a designer of an online course, I found it useful to use pre-test and post- 
test modules or units in order to measure learning outcomes of students." It 
is clear that the pre/post test phase of the assessment plan needs revision in 
FY96. Removing this question from the equation, the mean for the remaining 
four questions is 4.1. Instructor satisfaction with the online assessment 
process was positive. 

The third section of the questionnaire concentrated upon faculty 
training for online teaching. The overall mean for these six questions was 
3.8. Again, instructors were positive about the training process. However, 
five of the questions fell into the three range, which suggests that the 
online program in FY96 must concentrate more of its efforts into faculty 
training. 
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APPENDIX 8: NEHORANDUN RE PRE- AND POSHESTING ONLINE 



TO: Provost Powell 

FROM: Dennis Ridley and James Husband 
RE: Pre- and posttesting online 

DATE: August 8, 1995 



Pre- and posttesting of online course material was tried during spring 
1995 and also during the past summer. While the experience so far has been 
limited, both the data available for our inspection (spring, 1995) and the 
experience of this testing— from the standpoint of students and administrators 
alike — leads us to collaborate in making the following conclusions and 
recommendations. 

1. The major difficulty lies in student motivation. The task is 
ungraded and there is evidence that a number of students, particularly on the 
posttest, are not taking the task seriously. Zero percent correct on a 
posttest, for example, or other evidence of random responding, strongly 
indicate that student motivation is a problem. In some cases posttest results 
are poorer than pretest results. There was other evidence suggesting that the 
task students and faculty are being asked to support lacks face validity and 
does not fit well with the course experience. Rather than conclude that 
students are learning nothing, it is far easier to argue that lack of student 
effort is the source of the problem and that the data are largely invalid. 

2. Dr. Edward Smith who, as you know, recently conducted an 
"assessment of assessment" at CNU, recommended deleting the pretests. He 
pointed out that this change would reduce the workload and allow us to double 
the number of questions on the posttests and thereby increase their 
reliability. Furthermore, he found no compelling reason for using pretests 
since "the pertinent comparison is between the posttest scores for on-line vs. 
traditional sections." Our experience leads us to support Dr. Smith’s 
recommendation. 

3. Accordingly, we recommend that assessment testing be limited to 
course posttests, and that by spring semester of 1995 such tests be embedded 
in the course and be graded. We suggest that this change will ultimately 
become easier to administer, will enjoy greater faculty and student support, 
and wiir provide more valid assessment data. Details regarding test 
development, test content and the mechanics of test administration would 
remain to be worked out. We believe faculty should have wide latitude in 
regard to test content and format (e.g. essay vs. objective) as long as there 
existed credible external checks on the results. 

4. In the short term (for fall, 1995), we recommend that assessment 
testing should be limited to course posttesting but that the tests remain 
ungraded. As recommended by Dr. Smith, we support requiring longer and thus 
more valid posttests (about 25 multiple choice items). Jim has already 
approached several adjunct faculty members and received their consent to pilot 
test graded, embedded assessment testing. 

23 




61 



5. Finally, the credibility of the CNU Online program rests heavily 
on the demonstration of comparable learning online and in equivalent classroom 
courses. We urge that comparable testing of classroom sections should be made 
the rule rather than the exception. Jim has obtained tentative approval of 
several faculty members who are teaching comparable online and classroom 
sections to use identical posttest questions in the comparison classes using 
the embedded method. In addition, deans and department chairs need to become 
involved in order to facilitate these procedures. While some of these types 
of data have been collected, we need to strengthen the returns. 

We look forward to discussing these ideas with you on Thursday at 10:00 

a.m. 
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APPDDIX 9 



ONLINE/CLASSROON TEST COMPARISONS 

The 1995 CNU Online Assessment Plan called for each Instructor to 
develop test items for use in an online format. Pre-testing as well as post- 
testing was planned and was, in most cases, conducted. It is necessary to 
review that experience to account for the uses of testing for online/classroom 
comparisons. That experience was instructive and informative. However, one 
of the consequences of using pre- and post-testing with no grade associated 
with performance on these tests was a arguably a negative effect on the uses 
of tests for comparative purposes. Other circumstances, such as the approval 
of the Assessment Plan only days prior to the start of the spring semester, 
199Sj accounted for some confusion in the implementation of the plan. These 
circumstances also did not assist the effective use of tests for comparative 
purposes. 

The Assessment Plan called upon each online instructor to collaborate 
with a colleague who was teaching a classroom section of the same-numbered 
course. The two were required to agree on a core content, i.e., important 
concepts and terms that should be learned in the course. They were then 
required to agree on a set of questions that would be amenable to testing both 
online and in the classroom. (Of course, if the online instructor was also 
teaching the same course in the classroom the requirement could be met without 
involving another colleague.) However, due to the circumstances alluded to 
above, just four cases were found which minimally met this requirement. 

Because of varying circumstances, each case will be discussed separately. 

Case No. 1. This professor used both pre- and post-testing through two 
units of his course which he taught both online and in the classroom. Since 
pre- and posttests were not graded and were understood as such by students, it 
is probable that student motivation was low. In Case No. I, this 
interpretation would apply to both the online and classroom sections since the 
instructor, in the interest of consistency, used the same (ungraded) procedure 
for both classes. The evidence also indicates low motivation. For instance, 
a number of scores at or near zero on posttests strongly suggested random 
responding by a number of students. At the same time, other students received 
scores of 80 percent or higher. The mean scores, both online and standard 
courses, were between 31 and 44 percent. Online and classroom scores did not 
differ significantly from each other in a statistical sense. 

Case No. 2. This professor used the same unit tests in both online and 
classroom versions of the same course. The unit tests were averaged and the 
average grade was given substantial weight in determining grades. We may 
assume that student motivation was optimized by the knowledge that performance 
on the tests counted in the grade. Indeed, mean performance was over 70 
percent —much higher than in Case No. I. Although the online class 
consistently averaged higher than the classroom students, the difference was 
not statistically significant. 

Case No. 3. The professor was not able to implement comparative 
objective testing due to circumstances of late book arrivals and the necessity 
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of assigning different course material. However, the professor made use of 
some of the same essay questions on his final exams in the two courses. At 
the request of the Director of Assessment and Evaluation, he later addressed 
the issue of the comparability of the two sets of performances in a 
memorandum. He found that "...the work of the online students was at least as 
good, and I think better, than that of students in the traditional classroom." 
Further, he commented that the online students were forced out of a passive 
mode and "...they were by the end ready and able to make arguments in a 
critical way. I think their critical thinking skills were higher than the 
class average for the classroom section. 

Case No. 4. The professor used the same test and publisher’s test item 
bank as his colleague for the same introductory sociology course. The 
chapters and topics covered were quite similar. The colleague gave his 
students a final that was not comprehensive while the professor gave his 
students a comprehensive final from the same data bank source. While the 
colleague’s students’ test scores averaged somewhat higher, the difference was 
not statistically significantly. Assuming that the items averaged the same 
difficulty, and the online professor’s test was more challenging because it 
was comprehensive, the test performances of the two groups therefore were 
quite comparable. Horeover, the two groups received the same pre-test early 
in the course, the data of which suggested that the two classes started at a 
similar level of sophistication. 

The above four cases provide evidence which, though limited, supports 
the conclusion that when students in comparable classroom sections are tested 
using the same or similar tests and procedures, the online students perform 
comparably. There is some evidence (from Case No. 3) that the performance is 
superior when the test format calls for writing and a critical response. 
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APPENDIX 10 



CNU ONLINE STUDY OF PREREQUISITES, FALL TO SPRING 1994-95 



This study examined the question of how students performed In spring 
courses for which the prerequisites could be taken In the fall either online 
or in a traditional classroom setting. There were three courses offered in 
the fall in both modes which were prerequisite to courses offered in the 
spring: English 101, English 207G and Spanish 101. The corresponding courses 
offered in the spring were English 102, English 208G (not strictly a 
prerequisite, but the second course in a two-course sequence), and Spanish 
102 . 



Part I of this study looked at those who enrolled in the second course 
online. Given that the second course was taken online, how many enrolled in 
the prerequisite online versus in a classroom section? Further, how did 
students perform depending on whether they had taken the online or classroom 
version of the prerequisite? 

Out of 30 students who enrolled in English 102-70 in the spring, none 
(out of 5 who received a grade in 101-70) had taken English 101 online. All 
30 had taken it in the classroom. . 

Out of 26 students who enrolled in Spanish 102-70 in the spring, 17 or 
65.4 percent took the prerequisite in the classroom and 9 or 34.6 percent took 
it online. All 9 of those with the online prerequisite experience passed the 
spring online course, whereas only 6 out of the 17 of those with the classroom 
prerequisite passed. That difference was statistically significant with a 
probability of the results being due to chance equal to or less than 2 out of 
1000. 



Out of 13 students who enrolled in English 208G-70 in the spring, 9 or 
69.2 percent took the prerequisite in the classroom and 4 or 30.8 percent took 
it online. All 4 of those with online prerequisite experience passed the 
spring online course whereas only 6 of the 9 with, the classroom-based 
prerequisite experience passed. Again, that difference was statistically 
significant with a probability of a chance result equal to less than 5 out of 
100 . 



We may conclude that when sufficient numbers of fall online students 
return for a second online course in the spring to permit a statistical 
comparison with those whose prerequisites were taken in the classroom, 
performance in the course tends to favor those with the online experience. 

However, the interpretation of Part I by itself is ambiguous. It is 
unknown from these results alone how much the statistically better performance 
of the students with the online experience could be attributed to experience 
with the BBS system and online pedagogy versus the learning which presumably 
occurred through the online vehicle. Therefore, a Part II was logically 
required because it was essential to know how well the students with online 
experience (in the prerequisite) might do in a classroom course for which the 
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first course was prerequisite. Would these students* performance be 
comparable with traditionally taught students in the same course? 

However, due to the relative newness of CNU On-Line, the plan that was 
designed to provide an answer to the above question has so far failed to 
provide such an answer. Based on the two semesters (94-95) alone, only one 
person vfho had taken the prerequisite online enrolled for the second course in 
the classroom mode. While this person was not disadvantaged by the online 
prerequisite, one person could not provide enough data to draw a strong 
conclusion. As more terms of online experience accumulate, it will be 
possible to gather sufficient data to test whether students with the online 
experience perform comparably on a second (classroom-taught) course for which 
the online course was prerequisite. 

Nevertheless, while it is not yet possible to test by this method how 
well students have performed in traditional classes (as a function of online 
versus classroom-based prerequisites), another worthwhile observation may be 
made from this study. This obspjrvation concerns students’ enrollment choices. 
It is noteworthy that of the students with prior online experience, if they 
enrolled at all in the second course in the following semester, they tended to 
choose the online mode again. In fact, this pattern, noteworthy as it is, 
deprived us in this study of having sufficient numbers to observe the online 
students in a following classroom course to make an appropriate statistical 
test. 



In summary, students with onlire prerequisite experience tended to have 
an advantage in the second online course in a sequence over those students who 
did not have the online experience. Due to insufficient numbers it could not 
be ascertained whether students with online prerequisite experience would 
perform in a classroom course (the second in a sequence) at a level 
coonensurate with students with classroom prerequisite experience. This lack 
of numbers, however, was due to an apparent tendency for students who enrolled 
in a second course in a sequence to choose the online mode if they had taken 
and passed the first course in that mode. 
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APPENDIX 11 

SUMMARY OF PROFESSOR RONNIE COHEN*S 
REFLECTIONS ON TAKING INTRODUCTORY SPANISH 
BOTH ONLINE AND IN THE CLASSROOM, 1994-95 

Professor Ronnie Cohen has taken the introductory Spanish course both 
online (101) and in the classroom (102) and has offered several observations 
regarding the comparison between the two from the vantage of a student 
learner. She cautioned that some of her comments are course-specific; 
however, many points could be generalized to other courses and disciplines. 

She also cautioned that she is much more of a "professional learner" than our 
average student, and therefore both subject difficulty and motivation are 
different in her case. 

Professor Cohen first discussed what she regarded as distinct advantages 
of the online course compared to the classroom. The online experience puts a 
high premium on students becoming independent learners and sustaining 
independent learning. She described this experience as "immersion," a phrase 
which is particularly apt for the language experience. In the online course 
assignments in Spanish the learner was required to respond with whole 
sentences rather than bits of vocabulary or parts of speech such as might be 
elicited as responses in a face-to-face environment. Of necessity, the 
student had to type in a complete response, and this called for a type of 
immersion into the subject matter that Professor Cohen found relatively 
missing in the classroom course. The kind of performance which is ca'Hed for 
is much more demanding, and that is good from a learning standpoint. On the 
other hand, students in this situation find it extremely easy to fall behind 
and may not be able to catch up. 

Professor Cohen cautions that many students might find the "advantage" 
of the stress on independent learning to be a disadvantage in their 
perception. To such students, if they are asked to be independent it 
indicates that the teacher is not doing her job of teaching. 

However, if the students put in the effort required to keep up, they 
would find more individualized feedback and evaluation in the online course. 
This aspect of the online experience is something Professor Cohen described as 
a "big plus." Interestingly, Professor Cohen used oral speaking skills as an 
example of the benefits of individualized feedback, even though oral skills 
cannot be communicated directly over a computer bulletin board system. The 
Spanish professor (Cahill) had students read specific portions of text over a 
telephone into a recorder. Later each student received a detailed critique of 
her performance; more detailed and individualized than in the face-to-face 
classroom situation. In the latter setting it too often happened that some 
students could avoid participating in more than a perfunctory way, and in any 
case the feedback would be limited due to time constraints. 

Another big benefit of online classes according to Professor Cohen was 
the group work. First, as anyone who has ever taught can attest, students 
often can learn best by teaching others. Second, those receiving correction 
from another student can in turn also benefit by this individual attention 
from other persons beside the instructor. What happens in the latter case is 
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that students are often willing to take greater risks for the sake of learning 
in the non~evaluative context of being taught by peers. They are more willing 
to show what they don’t know, and thus they can be corrected in all the 
mistakes they are willing to reveal. Another aspect of group work is that it 
promotes an exchange of ideas that simply does not happen when the course 
assignments are all individualized. 

Another advantage that accrues to virtually all online courses is that 
of increasing students* skill and comfort level in using computers. Students 
who experience any phobia of computers or computer applications can hardly go 
through one of these courses. without substantially overcoming that handicap. 

Turning to actual or potential disadvantages, Professor Cohen made the 
following observations. 

1. It is extremely easy to get behind and not be able to catch up. This 
disadvantage is a consequence of the intense immersion in the subject matter 
that is required in online classes and the number of messages (not all 
noteworthy) that must be read. 

2. It is harder to find and exploit opportunities for conversation. In 
online courses, one has to work at making conversation happen— and therefore 
too the incidental learning that conversation can engender. 

3. Finally, not having personal contact with other students is, 
according to Professor Cohen, in the big picture ultimately a disadvantage. 
Communicating and learning in the digital environment can make a fine 
contribution to the University’s programs. It is only when this idea is 
extrapolated to a vision of totally substituting for, and replacing, the 
campus environment that Professor Cohen expresses reservations. In the real 
world, students also need to learn to function well in face-to-face 
situations; this is a goal that a digital learning environment cannot 
substantially help us achieve. 

Professor Cohen also responded to several criticisms, often levelled at 
computer-managed instruction, which she feels are weak or invalid. One 
criticism is that students engaging in group work can receive a free ride when 
they do not contribute their fair share of the work. While this certainly can 
occur, the same risk exists in group work in either context— online or in the 
classroom. Furthermore, the advantages of group work, including particularly 
the value of exchanging ideas with ones’ peers, outweigh this possible 
drawback. A second criticism is that because students do their work at a 
distance, it is easier to get away with turning in work which was done by 
somebody else. Professor Cohen argues that while this may be true, the 
difference between online courses and courses taught in the classroom has been 
exaggerated in this regard. Finding individuals who are both willing and 
capable of doing another’s work probably is not that easy. 

After having taught one course online. Prof. Cohen added that in her 
limited experience her learning was comparable to that of classroom courses, 
in some ways better and in some ways not as good. 
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